-- Logs begin at Wed 2022-10-19 20:18:53 UTC, end at Wed 2022-10-19 22:49:17 UTC. -- Oct 19 21:41:36.702180 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 19 21:41:36.710788 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: *** Starting uWSGI 2.0.18-debian (64bit) on [Wed Oct 19 21:41:36 2022] *** Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: nodename: dv-cinder-master-3-iscsi Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: machine: x86_64 Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: clock source: unix Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: pcre jit disabled Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: detected number of CPU cores: 4 Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: current working directory: / Oct 19 21:41:36.712878 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: detected binary path: /usr/bin/uwsgi-core Oct 19 21:41:36.713437 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: your processes number limit is 47680 Oct 19 21:41:36.713437 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: your memory page size is 4096 bytes Oct 19 21:41:36.713437 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: detected max file descriptor number: 2048 Oct 19 21:41:36.713437 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: lock engine: pthread robust mutexes Oct 19 21:41:36.713437 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: thunder lock: enabled Oct 19 21:41:36.713437 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 19 21:41:36.713437 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 19 21:41:36.752818 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: Python main interpreter initialized at 0x564b3dd9f9c0 Oct 19 21:41:36.752818 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: python threads support enabled Oct 19 21:41:36.752818 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: your server socket listen backlog is limited to 100 connections Oct 19 21:41:36.752818 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: your mercy for graceful operations on workers is 90 seconds Oct 19 21:41:36.753150 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: mapped 403077 bytes (393 KB) for 2 cores Oct 19 21:41:36.753224 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: *** Operational MODE: preforking *** Oct 19 21:41:36.753491 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: *** uWSGI is running in multiple interpreter mode *** Oct 19 21:41:36.753491 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: spawned uWSGI master process (pid: 83009) Oct 19 21:41:36.753685 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 19 21:41:36.758985 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: spawned uWSGI worker 1 (pid: 83021, cores: 1) Oct 19 21:41:36.758985 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: spawned uWSGI worker 2 (pid: 83022, cores: 1) Oct 19 21:41:36.758985 dv-cinder-master-3-iscsi devstack@keystone.service[83009]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 19 21:41:41.074835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 21:41:41.074835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: from cryptography.utils import int_from_bytes Oct 19 21:41:41.075602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 21:41:41.075602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: from cryptography.utils import int_from_bytes Oct 19 21:41:41.076025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 21:41:41.076025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: from cryptography.utils import int_from_bytes Oct 19 21:41:41.076134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 19 21:41:41.076134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: from cryptography.utils import int_from_bytes Oct 19 21:41:41.126361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 19 21:41:41.126553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 19 21:41:41.126746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] command line args: [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 19 21:41:41.127038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 19 21:41:41.127219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ================================================================================ {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 19 21:41:41.127472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] admin_token = **** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.127551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 19 21:41:41.127716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] config_dir = [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.127800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 19 21:41:41.127904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.127990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] command line args: [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 19 21:41:41.128088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] config_source = [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.128173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 19 21:41:41.128270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] debug = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.128354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ================================================================================ {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 19 21:41:41.128467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.128719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] admin_token = **** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.128836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] config_dir = [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.128918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.128984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] insecure_debug = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] config_source = [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] debug = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] list_limit = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] log_config_append = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.129994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] insecure_debug = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] log_dir = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] log_file = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] list_limit = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] log_config_append = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.130889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] log_dir = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] log_file = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131854 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.131975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.132926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.133028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.133157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.133245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.133402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.133486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.133646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] notification_format = cadf {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.133770 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.133916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.134039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.134166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.134414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] notification_format = cadf {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.134497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.20/identity {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.134611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] publish_errors = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.134697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.134842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.135097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.135199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.20/identity {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.139966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] publish_errors = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] standard_threads = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] strict_password_check = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.140910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] use_eventlog = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] standard_threads = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] use_journal = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141239 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] strict_password_check = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] use_json = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] use_stderr = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] use_syslog = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] use_eventlog = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] watch_log_file = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] use_journal = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] use_json = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.141959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] use_stderr = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.142106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] use_syslog = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.142253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] watch_log_file = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 19 21:41:41.142394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.142914 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143254 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143913 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.143983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144575 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.144945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.145967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.146999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147072 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.147906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148654 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.148960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.149076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.149151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.149271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.149347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.149498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.149587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.149920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.150856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.151008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.151095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.151169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.151296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.151373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.151433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.151547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.151631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152590 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.152939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.153950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.154967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.enabled = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.155879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.enabled = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.156971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157051 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.157984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.158139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.158217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.158377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.158463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.158613 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.158697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.158848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.158932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.159108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.159199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.159346 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.159426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.159586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.159665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.159821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.159908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.160981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.161981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.162123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.162202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.162362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.162443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.162597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.162683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.162844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.162925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.163103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.163183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.163361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] auth.external = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] auth.mapped = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] auth.external = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] auth.mapped = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] auth.password = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] auth.token = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.168983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] catalog.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] auth.password = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] auth.token = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] catalog.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169853 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.169923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] credential.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] credential.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] credential.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] credential.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.170937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.171965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] federation.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] federation.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] federation.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.172998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] federation.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173550 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173628 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.173990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.174861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175297 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.175922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.176008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.176293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.176387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.176468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.176542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.176667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.176798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.176912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.177983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.178150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.178236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.178401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.178478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.178638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.178724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.178870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.178983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.179127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.179224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.179357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.179435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.179586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.179674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.179821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.179899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.180055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.180137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.180292 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.180418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.180538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.180652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.180768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.180901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.181988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.182162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.182258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.182375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.182455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.182605 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.password = **** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.182687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.password = **** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.182841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.182920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.185859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.185971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.186942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.187999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.188941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.189940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190595 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.190931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191430 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191667 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.191992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.192147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.192225 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.192388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.192468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] policy.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.192620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.192698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] policy.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.192861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.192939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.193143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.193238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.193381 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] receipt.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.193468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.193616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] receipt.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.193694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.193853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.193932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.194091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.194168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.194323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.194402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.194554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.194631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.194791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.194868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] resource.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.195066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] resource.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.195151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.195319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.195400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] resource.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.195569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] resource.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.195647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.195806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.195884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.196056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.196136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.196302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] revoke.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.196378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.196535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] revoke.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.196612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.196770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.196847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] role.cache_time = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] role.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] role.cache_time = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] role.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] role.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] role.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] role.list_limit = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.197950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] role.list_limit = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.198027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.198199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.198278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.198442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.198519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.198713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.198791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.198971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.199109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.199231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.199436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.199523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.199690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.199777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.199942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.200989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.201158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.201240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.201550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.201628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.201788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.201865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.202038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.202117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.202288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.202365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.202537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.202615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.202785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.202863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.203042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.203121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.203300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.203389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.219388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.219480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.219554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.219642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.219748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.219824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.219903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.219976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220185 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] token.cache_time = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.220974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] token.cache_time = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] token.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] token.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] token.provider = fernet {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] token.provider = fernet {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221588 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.221969 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] trust.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] trust.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.222946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.connection = **** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.connection = **** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.223959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224575 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.224982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=83021) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 19 21:41:41.225769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 19 21:41:41.225769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=83022) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 19 21:41:41.244704 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.244959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.245169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.245394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.245672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.245895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.253544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.254448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.282086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.282316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.283443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.283676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.342020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.351100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.390420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.390650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.406938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.407194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.547283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.547515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.547784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548573 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.548900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.576559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.576795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.577943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.578014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.578095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.578302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.589186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.589754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.607185 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.607312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.607423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.607507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.607649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.607731 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.607874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.607964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.608095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.608176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.608311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.608393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.608528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=83021) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.608612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=83022) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 19 21:41:41.626167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.626371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.626457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 21:41:41.626587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 21:41:41.626708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.626906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 21:41:41.627009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.627161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 19 21:41:41.627242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.627599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.627695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.627770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.627842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.627959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.628050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.628224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.638305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.638450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.638581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.638734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.638855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.639031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.639210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.639320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.646208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.646442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.649110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.649393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.649476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.649915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.650014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.650205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.650286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.650486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.650607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.650778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.650874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.651070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.651153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.651343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.651423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.651614 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.651751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.652467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.652549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.652751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.652840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.653028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.676477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.676622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.677938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.678053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.678224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.678353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.683550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.683666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.683839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.683938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.684121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.684330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.690591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.690778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.690919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.691092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.695095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.695390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.695686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.696600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.696891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.697179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.697466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.697710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.697846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.698007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.698132 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.698276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.698399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.698576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.698711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.698875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.714436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.714581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.714708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.714845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.718562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.718692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.721675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.721821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.722093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.725173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.726313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.726441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.729505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.729650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.733243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.733546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.733837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.734104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.734238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.734408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.734526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.734694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.734815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.734996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.735318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.735642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.751442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.751737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.753776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.753849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.753924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.753996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.753996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.753996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.759077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.760136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.760764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.761735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.762355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.762639 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.763290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.763575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.765488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.765783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.766074 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.766302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.766385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.766586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.766667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.766869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.767176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.767568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.778317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.778606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.778896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.779137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.779258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.779421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.779502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.779700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.779788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.779975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.780055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.780351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.780640 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.780917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.796237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.797179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.799327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.800304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.802235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.803406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.803730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.804974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.805211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.805502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.805800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.806439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.806721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.807160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.812940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.814422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.814607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.816065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.817371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.817669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.817959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.818239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.818973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.819372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.819661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.819939 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.826292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.826599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.826896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.827942 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.828229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.828511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.833891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=83022) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.834174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.834461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.834742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.835053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.835336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.835547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=83021) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 19 21:41:41.835701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.835828 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.835964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.836102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.836232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.836358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.836482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.836647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.836775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.836920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.837049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.837196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.837330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.837457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.837639 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=83022) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.837768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.838038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.838380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.838661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.838929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.839808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=83021) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 19 21:41:41.872897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x564b3dd9f9c0 pid: 83022 (default app) Oct 19 21:41:41.874892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x564b3dd9f9c0 pid: 83021 (default app) Oct 19 21:41:41.876312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63c6d3-e51f-4c23-adb7-118828552ff4 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:41.876609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63c6d3-e51f-4c23-adb7-118828552ff4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:41.876888 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de63c6d3-e51f-4c23-adb7-118828552ff4 None None] PATH_INFO: `/v3` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:41.881095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1/1] 10.222.0.20 () {50 vars in 791 bytes} [Wed Oct 19 21:41:41 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:41:49.316623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6432518-6c46-4816-ad1e-f67632005649 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:49.317152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6432518-6c46-4816-ad1e-f67632005649 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:49.317152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6432518-6c46-4816-ad1e-f67632005649 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:49.321398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1/2] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:41:49 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:41:49.326967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:49.327377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:49.327524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:49.398370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: INFO dbcounter [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] Registered counter for database keystone Oct 19 21:41:49.418629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG oslo_db.sqlalchemy.engines [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=83022) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 19 21:41:49.425009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writer thread running {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 19 21:41:49.604079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG passlib.handlers.bcrypt [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=83022) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 19 21:41:49.604395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG passlib.handlers.bcrypt [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=83022) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 19 21:41:49.663178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:41:49.664786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:41:49.791626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 2/3] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:41:49 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 466 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 19 21:41:50.255527 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6432518-6c46-4816-ad1e-f67632005649 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:50.325630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: INFO dbcounter [None req-f6432518-6c46-4816-ad1e-f67632005649 None None] Registered counter for database keystone Oct 19 21:41:50.346884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG oslo_db.sqlalchemy.engines [None req-f6432518-6c46-4816-ad1e-f67632005649 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=83021) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 19 21:41:50.353089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writer thread running {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 19 21:41:50.476955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6432518-6c46-4816-ad1e-f67632005649 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:50.481794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e01da2-5fea-4c63-905f-6ace814ec1d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:50.482419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e01da2-5fea-4c63-905f-6ace814ec1d2 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:50.482697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e01da2-5fea-4c63-905f-6ace814ec1d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:50.482981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e01da2-5fea-4c63-905f-6ace814ec1d2 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:50.491027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e01da2-5fea-4c63-905f-6ace814ec1d2 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:50.544537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING keystone.common.rbac_enforcer.enforcer [None req-70e01da2-5fea-4c63-905f-6ace814ec1d2 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 19 21:41:50.544835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e01da2-5fea-4c63-905f-6ace814ec1d2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:50.553951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 2/4] 10.222.0.20 () {58 vars in 1324 bytes} [Wed Oct 19 21:41:50 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 21:41:50.559893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:50.576728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0710bd-bcf7-4d69-94b7-b9309ae1cba9 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:50.581068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00526d2-5ad2-4abd-aba1-54c62f272578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:50.582150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00526d2-5ad2-4abd-aba1-54c62f272578 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:50.582423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00526d2-5ad2-4abd-aba1-54c62f272578 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:50.582688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00526d2-5ad2-4abd-aba1-54c62f272578 admin admin] PATH_INFO: `/v3/projects/0fd8fb8e58c74c8cbe244194ce71b304` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:50.584308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00526d2-5ad2-4abd-aba1-54c62f272578 admin admin] RBAC: Authorizing `identity:get_project(project_id=0fd8fb8e58c74c8cbe244194ce71b304)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:50.638521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING keystone.common.rbac_enforcer.enforcer [None req-e00526d2-5ad2-4abd-aba1-54c62f272578 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 19 21:41:50.638867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00526d2-5ad2-4abd-aba1-54c62f272578 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:50.643356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 3/5] 10.222.0.20 () {56 vars in 1207 bytes} [Wed Oct 19 21:41:50 2022] GET /identity/v3/projects/0fd8fb8e58c74c8cbe244194ce71b304 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:41:52.745973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e2f2a-9d07-438e-bd05-15df3a5aadfc None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:52.746492 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e2f2a-9d07-438e-bd05-15df3a5aadfc None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:52.746603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7e2f2a-9d07-438e-bd05-15df3a5aadfc None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:52.750684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 3/6] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:41:52 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:41:52.756197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ed8f6-9d7b-4ae4-916d-75c57da81ee3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:52.756566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ed8f6-9d7b-4ae4-916d-75c57da81ee3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:52.756760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-082ed8f6-9d7b-4ae4-916d-75c57da81ee3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:52.777919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-082ed8f6-9d7b-4ae4-916d-75c57da81ee3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:41:52.779487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-082ed8f6-9d7b-4ae4-916d-75c57da81ee3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:41:52.787428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 4/7] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:41:52 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 21:41:53.051523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e2f2a-9d07-438e-bd05-15df3a5aadfc None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:53.064058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7e2f2a-9d07-438e-bd05-15df3a5aadfc None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:53.068308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfce821-6a7c-40ee-ac0f-28b42a200af7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:53.068902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfce821-6a7c-40ee-ac0f-28b42a200af7 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:53.069185 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfce821-6a7c-40ee-ac0f-28b42a200af7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:53.069461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfce821-6a7c-40ee-ac0f-28b42a200af7 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:53.077063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfce821-6a7c-40ee-ac0f-28b42a200af7 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:53.078898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cfce821-6a7c-40ee-ac0f-28b42a200af7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:53.087541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 4/8] 10.222.0.20 () {58 vars in 1324 bytes} [Wed Oct 19 21:41:53 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 21:41:53.093448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ed8f6-9d7b-4ae4-916d-75c57da81ee3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:53.102522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-082ed8f6-9d7b-4ae4-916d-75c57da81ee3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:53.107482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f03f216-23a7-421d-b370-5b2fdaeb2abb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:53.108098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f03f216-23a7-421d-b370-5b2fdaeb2abb admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:53.108384 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f03f216-23a7-421d-b370-5b2fdaeb2abb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:53.108657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f03f216-23a7-421d-b370-5b2fdaeb2abb admin admin] PATH_INFO: `/v3/users/4cc834c20ef040c19350af5fc47f4735` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:53.109960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f03f216-23a7-421d-b370-5b2fdaeb2abb admin admin] RBAC: Authorizing `identity:get_user(user_id=4cc834c20ef040c19350af5fc47f4735)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:53.111827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f03f216-23a7-421d-b370-5b2fdaeb2abb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:53.113339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 5/9] 10.222.0.20 () {56 vars in 1198 bytes} [Wed Oct 19 21:41:53 2022] GET /identity/v3/users/4cc834c20ef040c19350af5fc47f4735 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:41:55.289522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025eb65-d64c-4334-ba38-9665e8e5b2ea None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:55.290037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025eb65-d64c-4334-ba38-9665e8e5b2ea None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:55.290145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5025eb65-d64c-4334-ba38-9665e8e5b2ea None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:55.291423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 5/10] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:41:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:41:55.296856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db42ec-25de-451c-897c-88e24d656c36 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:55.297247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db42ec-25de-451c-897c-88e24d656c36 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:55.297410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db42ec-25de-451c-897c-88e24d656c36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:55.315048 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b5db42ec-25de-451c-897c-88e24d656c36 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:41:55.316238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b5db42ec-25de-451c-897c-88e24d656c36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:41:55.363255 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e03538a-3c1d-45a0-829c-fecd6e3797c8 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:55.363532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e03538a-3c1d-45a0-829c-fecd6e3797c8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:55.363803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e03538a-3c1d-45a0-829c-fecd6e3797c8 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:55.365171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 6/11] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:41:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:41:55.370133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:55.371166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:55.371435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:55.397875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 6/12] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:41:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:41:55.413363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG passlib.handlers.bcrypt [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=83021) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 19 21:41:55.413764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG passlib.handlers.bcrypt [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=83021) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 19 21:41:55.463041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:41:55.464637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:41:55.562171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 7/13] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:41:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:41:55.662859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5db42ec-25de-451c-897c-88e24d656c36 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:55.688273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:41:55.768587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5db42ec-25de-451c-897c-88e24d656c36 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:55.802197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2624dd52-bd6e-4c9f-866e-4b93b52b8b89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:55.802833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2624dd52-bd6e-4c9f-866e-4b93b52b8b89 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:55.803140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2624dd52-bd6e-4c9f-866e-4b93b52b8b89 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:55.803415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2624dd52-bd6e-4c9f-866e-4b93b52b8b89 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:55.814365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2624dd52-bd6e-4c9f-866e-4b93b52b8b89 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-2624dd52-bd6e-4c9f-866e-4b93b52b8b89 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:41:55.823208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:41:55.824545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:41:55.825444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 7/14] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:41:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:41:55.827737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:55.830349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2624dd52-bd6e-4c9f-866e-4b93b52b8b89 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:55.912711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:41:55.929572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0023225b-2470-4da1-9e9b-c8741416065a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:55.930173 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2624dd52-bd6e-4c9f-866e-4b93b52b8b89 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:55.931821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:41:55.965425 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:55.965775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:55.966027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:55.966307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:55.966580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:55.966663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:55.966903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:55.967204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:55.967324 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:41:55.967630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:55.968182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:55.969039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:55.969603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:55.980766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 8/15] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:41:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:41:55.986356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:56.000170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.sql.core [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 21:41:56.000170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 21:41:56.000170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [parameters: {'id': 'aa8085245d554a898d8943ab84e59798', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 21:41:56.000170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=83021) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:41:56.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result.read() Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 21:41:56.021206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application yield resource Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise self.value Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application session.commit() Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 21:41:56.022272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.session.flush() Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application compat.raise_( Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 21:41:56.023432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 21:41:56.024288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result.read() Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:41:56.024983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [parameters: {'id': 'aa8085245d554a898d8943ab84e59798', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 21:41:56.026023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:41:56.027547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:41:56.028565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 8/16] 10.222.0.20 () {60 vars in 1137 bytes} [Wed Oct 19 21:41:55 2022] POST /identity/v3/domains => generated 190 bytes in 194 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:41:56.029329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:56.088812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746dcee8-43db-4b7a-9857-30fa4df183cb None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:56.121655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec3434f-14d6-417a-b884-f276c21cc58c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:56.122264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3434f-14d6-417a-b884-f276c21cc58c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:56.122563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3434f-14d6-417a-b884-f276c21cc58c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:56.122842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec3434f-14d6-417a-b884-f276c21cc58c None admin] PATH_INFO: `/v3/users/4cc834c20ef040c19350af5fc47f4735` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:56.123999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec3434f-14d6-417a-b884-f276c21cc58c None admin] RBAC: Authorizing `identity:get_user(user_id=4cc834c20ef040c19350af5fc47f4735)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:56.125452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec3434f-14d6-417a-b884-f276c21cc58c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:56.127009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 9/17] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:41:55 2022] GET /identity/v3/users/4cc834c20ef040c19350af5fc47f4735 => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:41:56.128459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e7a762-a2d3-4f28-912e-7f7ce5b02ddd None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:56.132563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec3434f-14d6-417a-b884-f276c21cc58c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:56.147673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:41:56.163227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:56.164102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:56.167318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:56.167318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:56.181997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:56.184036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:41:56.200022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:41:56.201024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:41:56.202077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:41:56.202077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:41:56.202077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:41:56.202077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 9/18] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:41:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:41:56.206839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:56.227720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:41:56.232666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec3434f-14d6-417a-b884-f276c21cc58c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:56.266599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f1442a-22bc-443a-bfd1-a1798c891ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:56.267273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f1442a-22bc-443a-bfd1-a1798c891ba0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:56.267604 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f1442a-22bc-443a-bfd1-a1798c891ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:56.267891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f1442a-22bc-443a-bfd1-a1798c891ba0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:56.269179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f1442a-22bc-443a-bfd1-a1798c891ba0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:56.270653 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f1442a-22bc-443a-bfd1-a1798c891ba0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:56.272243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 10/19] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:41:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:41:56.278533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f1442a-22bc-443a-bfd1-a1798c891ba0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:56.306334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab060e6-1171-4aab-bb5d-1eb6cff457b9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:56.342257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6332294-06aa-4078-bde3-d1408a1a755e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:56.342788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6332294-06aa-4078-bde3-d1408a1a755e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:56.343101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6332294-06aa-4078-bde3-d1408a1a755e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:56.343375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6332294-06aa-4078-bde3-d1408a1a755e None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:56.344012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6332294-06aa-4078-bde3-d1408a1a755e None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:41:56.344315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6332294-06aa-4078-bde3-d1408a1a755e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:56.345695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6332294-06aa-4078-bde3-d1408a1a755e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:56.358275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:41:56.362929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 10/20] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:41:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:41:56.378269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f1442a-22bc-443a-bfd1-a1798c891ba0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:56.414492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd51bae-7609-478a-95df-d91b6f341143 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:56.415206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd51bae-7609-478a-95df-d91b6f341143 None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:56.415488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd51bae-7609-478a-95df-d91b6f341143 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:56.415766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd51bae-7609-478a-95df-d91b6f341143 None admin] PATH_INFO: `/v3/domains/default/users/4cc834c20ef040c19350af5fc47f4735/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:56.418393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd51bae-7609-478a-95df-d91b6f341143 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=4cc834c20ef040c19350af5fc47f4735, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:41:56.419944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd51bae-7609-478a-95df-d91b6f341143 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:41:56.608833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 11/21] 10.222.0.20 () {58 vars in 1361 bytes} [Wed Oct 19 21:41:56 2022] PUT /identity/v3/domains/default/users/4cc834c20ef040c19350af5fc47f4735/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 21:41:59.073822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d7662e-788a-41b2-baa4-53e92942c22e None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.074344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d7662e-788a-41b2-baa4-53e92942c22e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.074344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d7662e-788a-41b2-baa4-53e92942c22e None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.076049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 11/22] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:41:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:41:59.081893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a7d19d-dde0-4a2d-b0ba-77ec73722f26 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.082309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a7d19d-dde0-4a2d-b0ba-77ec73722f26 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.082529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a7d19d-dde0-4a2d-b0ba-77ec73722f26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.103629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-48a7d19d-dde0-4a2d-b0ba-77ec73722f26 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:41:59.105894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-48a7d19d-dde0-4a2d-b0ba-77ec73722f26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:41:59.139778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f6e655-bc3b-496e-aff1-c71913f3cca8 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.140383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f6e655-bc3b-496e-aff1-c71913f3cca8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.140853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f6e655-bc3b-496e-aff1-c71913f3cca8 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.142576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 12/23] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:41:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:41:59.148025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3535d0af-10dc-4c86-a011-713c83fd25a6 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.148565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3535d0af-10dc-4c86-a011-713c83fd25a6 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.149246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3535d0af-10dc-4c86-a011-713c83fd25a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.184350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3535d0af-10dc-4c86-a011-713c83fd25a6 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:41:59.185584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3535d0af-10dc-4c86-a011-713c83fd25a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:41:59.221612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 12/24] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:41:59.301117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 13/25] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:41:59.368637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759e740a-f772-4d58-ac91-d8968d301cf1 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.368917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759e740a-f772-4d58-ac91-d8968d301cf1 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.369125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759e740a-f772-4d58-ac91-d8968d301cf1 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.374216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 13/26] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:41:59 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:41:59.388924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf71a22e-3f22-4680-9071-8369e087e616 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.389485 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf71a22e-3f22-4680-9071-8369e087e616 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.390076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf71a22e-3f22-4680-9071-8369e087e616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.412240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-cf71a22e-3f22-4680-9071-8369e087e616 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:41:59.413724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cf71a22e-3f22-4680-9071-8369e087e616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:41:59.416902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:41:59.498974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 14/27] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:41:59.642183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759e740a-f772-4d58-ac91-d8968d301cf1 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:59.739870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:41:59.744178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83fad63-48a3-48c2-8065-4fc9346e1030 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.744717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83fad63-48a3-48c2-8065-4fc9346e1030 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.745198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83fad63-48a3-48c2-8065-4fc9346e1030 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.747642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 15/28] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:41:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:41:59.767251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc7a6d-f905-4446-9f74-f0eb62e95ded None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.767852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc7a6d-f905-4446-9f74-f0eb62e95ded None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.768325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9dc7a6d-f905-4446-9f74-f0eb62e95ded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.784701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759e740a-f772-4d58-ac91-d8968d301cf1 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:41:59.808897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d9dc7a6d-f905-4446-9f74-f0eb62e95ded None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:41:59.811126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d9dc7a6d-f905-4446-9f74-f0eb62e95ded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:41:59.857552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5de0f2-b2cd-4429-a8bf-133f65d6ef18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:41:59.858550 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5de0f2-b2cd-4429-a8bf-133f65d6ef18 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:41:59.859061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5de0f2-b2cd-4429-a8bf-133f65d6ef18 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:41:59.859528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5de0f2-b2cd-4429-a8bf-133f65d6ef18 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:41:59.877539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5de0f2-b2cd-4429-a8bf-133f65d6ef18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-2f5de0f2-b2cd-4429-a8bf-133f65d6ef18 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:41:59.885053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:41:59.885941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:41:59.886741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 14/29] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:41:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:41:59.887419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5de0f2-b2cd-4429-a8bf-133f65d6ef18 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:41:59.903246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 16/30] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:41:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:41:59.904528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dc7a6d-f905-4446-9f74-f0eb62e95ded None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:00.037525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:00.057223 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5de0f2-b2cd-4429-a8bf-133f65d6ef18 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:00.092872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9dc7a6d-f905-4446-9f74-f0eb62e95ded None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:00.114272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:00.125708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d59dd8-7dd7-492d-a3a2-757158e0e6b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:00.130365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d59dd8-7dd7-492d-a3a2-757158e0e6b8 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.130365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d59dd8-7dd7-492d-a3a2-757158e0e6b8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.130365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d59dd8-7dd7-492d-a3a2-757158e0e6b8 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.130365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d59dd8-7dd7-492d-a3a2-757158e0e6b8 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:00.131994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d59dd8-7dd7-492d-a3a2-757158e0e6b8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:00.158994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:00.158994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.159419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.159946 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.172435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:00.174371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:42:00.196312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:00.197254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:42:00.198040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:42:00.198040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:00.198040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:00.198040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 17/31] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:41:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:00.200551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:00.246866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 15/32] 10.222.0.20 () {60 vars in 1131 bytes} [Wed Oct 19 21:41:59 2022] POST /identity/v3/roles => generated 219 bytes in 360 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:00.248203 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d59dd8-7dd7-492d-a3a2-757158e0e6b8 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:00.315657 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7823a88-8b42-44dd-a750-6681ca30926e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:00.341293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:00.354078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:00.355357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.355644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.355916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.356630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] RBAC: Adding query filter params (name=admin) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:00.356937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:00.358386 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:00.372124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 18/33] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:00.372986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:00.407322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d59dd8-7dd7-492d-a3a2-757158e0e6b8 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:00.449303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6a2454-7945-418c-b025-614f8df6c5ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:00.450140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a2454-7945-418c-b025-614f8df6c5ee None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.450260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a2454-7945-418c-b025-614f8df6c5ee None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.450487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6a2454-7945-418c-b025-614f8df6c5ee None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.451201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6a2454-7945-418c-b025-614f8df6c5ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:00.451521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6a2454-7945-418c-b025-614f8df6c5ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:00.453152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6a2454-7945-418c-b025-614f8df6c5ee None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:00.478053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 16/34] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:42:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:00.481527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13013f84-d08d-4828-9c23-ebe8bb83a635 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.482040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13013f84-d08d-4828-9c23-ebe8bb83a635 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.482540 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13013f84-d08d-4828-9c23-ebe8bb83a635 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.486543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 17/35] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:00 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:00.488569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13013f84-d08d-4828-9c23-ebe8bb83a635 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:00.511207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475ecc2c-72dc-4d85-bd0d-4d26ef92192a None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:00.552163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:00.553845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3610197d-a73e-4190-a7e7-dee72529d8f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:00.554459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610197d-a73e-4190-a7e7-dee72529d8f5 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.554738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610197d-a73e-4190-a7e7-dee72529d8f5 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.555037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610197d-a73e-4190-a7e7-dee72529d8f5 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.556127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3610197d-a73e-4190-a7e7-dee72529d8f5 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:00.557826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3610197d-a73e-4190-a7e7-dee72529d8f5 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:00.604689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13013f84-d08d-4828-9c23-ebe8bb83a635 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:00.638289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea78d91-54b9-4abd-bb84-c955e22b2676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:00.639050 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea78d91-54b9-4abd-bb84-c955e22b2676 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.639912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea78d91-54b9-4abd-bb84-c955e22b2676 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.640573 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea78d91-54b9-4abd-bb84-c955e22b2676 None admin] PATH_INFO: `/v3/users/4cc834c20ef040c19350af5fc47f4735` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.642661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea78d91-54b9-4abd-bb84-c955e22b2676 None admin] RBAC: Authorizing `identity:get_user(user_id=4cc834c20ef040c19350af5fc47f4735)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:00.644208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea78d91-54b9-4abd-bb84-c955e22b2676 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:00.645905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 18/36] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:00 2022] GET /identity/v3/users/4cc834c20ef040c19350af5fc47f4735 => generated 247 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:00.646689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea78d91-54b9-4abd-bb84-c955e22b2676 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:00.763089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 19/37] 10.222.0.20 () {60 vars in 1131 bytes} [Wed Oct 19 21:42:00 2022] POST /identity/v3/roles => generated 225 bytes in 383 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:00.763089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d684ad3b-3180-42d4-8f6c-8695cb6e0e8d None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.763089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d684ad3b-3180-42d4-8f6c-8695cb6e0e8d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.763089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d684ad3b-3180-42d4-8f6c-8695cb6e0e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.788618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d684ad3b-3180-42d4-8f6c-8695cb6e0e8d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:00.790869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d684ad3b-3180-42d4-8f6c-8695cb6e0e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:00.840157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:00.870347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea78d91-54b9-4abd-bb84-c955e22b2676 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:00.883179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 20/38] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:00.883179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d684ad3b-3180-42d4-8f6c-8695cb6e0e8d None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:00.891042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:00.912087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a6bb9b-0ead-41d0-9965-a779b45a17ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:00.912890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6bb9b-0ead-41d0-9965-a779b45a17ab None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:00.913310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6bb9b-0ead-41d0-9965-a779b45a17ab None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:00.913991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6bb9b-0ead-41d0-9965-a779b45a17ab None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:00.915069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a6bb9b-0ead-41d0-9965-a779b45a17ab None admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:00.917896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a6bb9b-0ead-41d0-9965-a779b45a17ab None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:01.006792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d684ad3b-3180-42d4-8f6c-8695cb6e0e8d None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:01.014077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 19/39] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:42:00 2022] POST /identity/v3/projects => generated 301 bytes in 367 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:01.015302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c73a1-b72d-45b5-8a11-0b0d2dcc27b0 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:01.015598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c73a1-b72d-45b5-8a11-0b0d2dcc27b0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:01.016016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c73a1-b72d-45b5-8a11-0b0d2dcc27b0 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:01.017410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 20/40] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:01.023475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829e1f8-65d9-4d28-a1f8-26e2fe77f04a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:01.023827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829e1f8-65d9-4d28-a1f8-26e2fe77f04a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:01.024115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829e1f8-65d9-4d28-a1f8-26e2fe77f04a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:01.045192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-9829e1f8-65d9-4d28-a1f8-26e2fe77f04a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:01.046511 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9829e1f8-65d9-4d28-a1f8-26e2fe77f04a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:01.052771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccf721f-4f29-43aa-aff7-648f3c29cb4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:01.053897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccf721f-4f29-43aa-aff7-648f3c29cb4c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:01.054395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccf721f-4f29-43aa-aff7-648f3c29cb4c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:01.054877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccf721f-4f29-43aa-aff7-648f3c29cb4c None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:01.056686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccf721f-4f29-43aa-aff7-648f3c29cb4c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:01.058882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccf721f-4f29-43aa-aff7-648f3c29cb4c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:01.061333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 21/41] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:01.069373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccf721f-4f29-43aa-aff7-648f3c29cb4c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:01.132087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 21/42] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:01.136879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:01.156824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9829e1f8-65d9-4d28-a1f8-26e2fe77f04a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:01.208447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccf721f-4f29-43aa-aff7-648f3c29cb4c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:01.259235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:01.260013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:01.260301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:01.260749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:01.261597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.domain.id=default, user.id=4cc834c20ef040c19350af5fc47f4735) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:01.261904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:01.263458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:01.283078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 22/43] 10.222.0.20 () {56 vars in 1322 bytes} [Wed Oct 19 21:42:01 2022] GET /identity/v3/role_assignments?user.id=4cc834c20ef040c19350af5fc47f4735&scope.domain.id=default&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 530 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:01.291622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:01.295627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9829e1f8-65d9-4d28-a1f8-26e2fe77f04a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:01.337716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e914a702-8b77-49de-8232-8ddeca6e72ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:01.338641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e914a702-8b77-49de-8232-8ddeca6e72ff None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:01.339755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e914a702-8b77-49de-8232-8ddeca6e72ff None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:01.340178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e914a702-8b77-49de-8232-8ddeca6e72ff None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:01.341050 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e914a702-8b77-49de-8232-8ddeca6e72ff None admin] RBAC: Authorizing `identity:create_role()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:01.342760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e914a702-8b77-49de-8232-8ddeca6e72ff None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:01.420862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:01.452884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 22/44] 10.222.0.20 () {60 vars in 1131 bytes} [Wed Oct 19 21:42:01 2022] POST /identity/v3/roles => generated 223 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:42:01.535258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc8c216-39b4-418e-b3bd-db51f122bcc2 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:01.543581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:01.571358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c54174e-6b99-4619-8bc3-5d88e932782d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:01.572022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c54174e-6b99-4619-8bc3-5d88e932782d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:01.572309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c54174e-6b99-4619-8bc3-5d88e932782d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:01.572588 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c54174e-6b99-4619-8bc3-5d88e932782d None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:01.574357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c54174e-6b99-4619-8bc3-5d88e932782d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:01.575945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c54174e-6b99-4619-8bc3-5d88e932782d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:01.579222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 23/45] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:01 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:01.583598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e914a702-8b77-49de-8232-8ddeca6e72ff None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:01.687932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e914a702-8b77-49de-8232-8ddeca6e72ff None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:01.720872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373656fe-434b-45da-bd10-2d6feb9206fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:01.721478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373656fe-434b-45da-bd10-2d6feb9206fc None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:01.721760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373656fe-434b-45da-bd10-2d6feb9206fc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:01.722037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373656fe-434b-45da-bd10-2d6feb9206fc None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:01.722698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373656fe-434b-45da-bd10-2d6feb9206fc None admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:01.724203 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373656fe-434b-45da-bd10-2d6feb9206fc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:01.789929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 23/46] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:42:01 2022] POST /identity/v3/projects => generated 312 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:03.926651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc92215-fb81-488a-87f7-aa6801f5563e None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:03.927174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc92215-fb81-488a-87f7-aa6801f5563e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:03.927256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc92215-fb81-488a-87f7-aa6801f5563e None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:03.928587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 24/47] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:03.933866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2d5b5b-084d-49c6-9d61-b743e7972f16 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:03.934274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2d5b5b-084d-49c6-9d61-b743e7972f16 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:03.934420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2d5b5b-084d-49c6-9d61-b743e7972f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:03.952412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6b2d5b5b-084d-49c6-9d61-b743e7972f16 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:03.953620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6b2d5b5b-084d-49c6-9d61-b743e7972f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:03.960769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:04.018583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 24/48] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:04.287074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc92215-fb81-488a-87f7-aa6801f5563e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:04.387145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc92215-fb81-488a-87f7-aa6801f5563e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:04.419494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7a359b-e6ac-46ca-8624-1949209ce57d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:04.420111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a359b-e6ac-46ca-8624-1949209ce57d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:04.420396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a359b-e6ac-46ca-8624-1949209ce57d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:04.420673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a359b-e6ac-46ca-8624-1949209ce57d None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:04.421849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7a359b-e6ac-46ca-8624-1949209ce57d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:04.423342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7a359b-e6ac-46ca-8624-1949209ce57d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:04.424947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 25/49] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:04.430687 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2d5b5b-084d-49c6-9d61-b743e7972f16 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:04.529227 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b2d5b5b-084d-49c6-9d61-b743e7972f16 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:04.567010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad4b226-71c5-4a3f-a633-ed12a0bd0340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:04.567565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4b226-71c5-4a3f-a633-ed12a0bd0340 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:04.567848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4b226-71c5-4a3f-a633-ed12a0bd0340 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:04.568125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad4b226-71c5-4a3f-a633-ed12a0bd0340 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:04.568786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad4b226-71c5-4a3f-a633-ed12a0bd0340 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:04.570246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aad4b226-71c5-4a3f-a633-ed12a0bd0340 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:04.584725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:04.743057 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 25/50] 10.222.0.20 () {60 vars in 1140 bytes} [Wed Oct 19 21:42:04 2022] POST /identity/v3/projects => generated 298 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:06.885694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7ee6fb-2d6a-420a-9db2-a00fc6cecc0b None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:06.886245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7ee6fb-2d6a-420a-9db2-a00fc6cecc0b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:06.886245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7ee6fb-2d6a-420a-9db2-a00fc6cecc0b None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:06.887600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 26/51] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:06.892752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4a0ad-8590-4c39-9b17-c8495e83f4ac None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:06.893164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4a0ad-8590-4c39-9b17-c8495e83f4ac None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:06.893322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c4a0ad-8590-4c39-9b17-c8495e83f4ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:06.912012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-23c4a0ad-8590-4c39-9b17-c8495e83f4ac None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:06.913203 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-23c4a0ad-8590-4c39-9b17-c8495e83f4ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:06.977372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 26/52] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:07.244760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7ee6fb-2d6a-420a-9db2-a00fc6cecc0b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:07.288126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:07.344407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b7ee6fb-2d6a-420a-9db2-a00fc6cecc0b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:07.377055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63050d4e-f423-4017-8c1f-8e1eb28cf6fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:07.377676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63050d4e-f423-4017-8c1f-8e1eb28cf6fe None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:07.377961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63050d4e-f423-4017-8c1f-8e1eb28cf6fe None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:07.378240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63050d4e-f423-4017-8c1f-8e1eb28cf6fe None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:07.379384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63050d4e-f423-4017-8c1f-8e1eb28cf6fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:07.380914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63050d4e-f423-4017-8c1f-8e1eb28cf6fe None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:07.382512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 27/53] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:07.388670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c4a0ad-8590-4c39-9b17-c8495e83f4ac None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:07.484330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c4a0ad-8590-4c39-9b17-c8495e83f4ac None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:07.507199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:07.517971 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9db2c-6af0-40bf-959d-3f30628a5fda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:07.518574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9db2c-6af0-40bf-959d-3f30628a5fda None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:07.518882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9db2c-6af0-40bf-959d-3f30628a5fda None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:07.519166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9db2c-6af0-40bf-959d-3f30628a5fda None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:07.519824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9db2c-6af0-40bf-959d-3f30628a5fda None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:07.521280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9db2c-6af0-40bf-959d-3f30628a5fda None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:07.605059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 27/54] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:07 2022] POST /identity/v3/users => generated 275 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:10.814348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aa9999-2192-4aa5-8e8a-da1367aa31a8 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:10.815372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aa9999-2192-4aa5-8e8a-da1367aa31a8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:10.815893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aa9999-2192-4aa5-8e8a-da1367aa31a8 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:10.817707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 28/55] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:10.823185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcef165-8cf3-4c5f-9f3f-130c3ce74bfa None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:10.824088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcef165-8cf3-4c5f-9f3f-130c3ce74bfa None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:10.824561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcef165-8cf3-4c5f-9f3f-130c3ce74bfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:10.847544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8bcef165-8cf3-4c5f-9f3f-130c3ce74bfa None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:10.848352 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127621cf-8467-4604-a6b8-52e5af767f41 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:10.848758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8bcef165-8cf3-4c5f-9f3f-130c3ce74bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:10.849082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127621cf-8467-4604-a6b8-52e5af767f41 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:10.849544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127621cf-8467-4604-a6b8-52e5af767f41 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:10.851089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 29/56] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:42:10.856925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571c6df-0a70-4636-bb74-6d001163842f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:10.857481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571c6df-0a70-4636-bb74-6d001163842f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:10.857946 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f571c6df-0a70-4636-bb74-6d001163842f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:10.891951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f571c6df-0a70-4636-bb74-6d001163842f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:10.893158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f571c6df-0a70-4636-bb74-6d001163842f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:10.962530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 28/57] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:11.018460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 30/58] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:11.117917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa975cc-396e-4ad8-a088-af10d009ba74 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.118515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa975cc-396e-4ad8-a088-af10d009ba74 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.119047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa975cc-396e-4ad8-a088-af10d009ba74 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.121253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 29/59] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:11.126537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870813fc-d256-447c-8425-e15af399841a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.127087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870813fc-d256-447c-8425-e15af399841a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.127565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870813fc-d256-447c-8425-e15af399841a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.151660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-870813fc-d256-447c-8425-e15af399841a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:11.153202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-870813fc-d256-447c-8425-e15af399841a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:11.191992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:11.232385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 31/60] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:11.287419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa975cc-396e-4ad8-a088-af10d009ba74 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:11.293605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870813fc-d256-447c-8425-e15af399841a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:11.444368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa975cc-396e-4ad8-a088-af10d009ba74 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:11.446087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870813fc-d256-447c-8425-e15af399841a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:11.457234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:11.484568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bfbc62-ab1e-4c62-86a0-aedb64c70d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:11.485227 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bfbc62-ab1e-4c62-86a0-aedb64c70d77 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.485511 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bfbc62-ab1e-4c62-86a0-aedb64c70d77 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.485784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bfbc62-ab1e-4c62-86a0-aedb64c70d77 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.497966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bfbc62-ab1e-4c62-86a0-aedb64c70d77 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-71bfbc62-ab1e-4c62-86a0-aedb64c70d77 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:11.503395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:11.504308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:11.505031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 30/61] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:42:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:11.512053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c0577-3faf-4244-88b3-417dc1c66f58 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.512360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c0577-3faf-4244-88b3-417dc1c66f58 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.512689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42c0577-3faf-4244-88b3-417dc1c66f58 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.519470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 31/62] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:11 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:11.519470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42c0577-3faf-4244-88b3-417dc1c66f58 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:11.611892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c11dbc-0325-4928-8bea-564dfb7f9d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:11.612522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c11dbc-0325-4928-8bea-564dfb7f9d2d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.612806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c11dbc-0325-4928-8bea-564dfb7f9d2d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.613068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c11dbc-0325-4928-8bea-564dfb7f9d2d None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.625116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c11dbc-0325-4928-8bea-564dfb7f9d2d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:11.629149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42c0577-3faf-4244-88b3-417dc1c66f58 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-a8c11dbc-0325-4928-8bea-564dfb7f9d2d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:11.632104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:11.633011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 32/63] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 340 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:11.633786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7eefe5-6f6e-47ac-8622-02b2edb38ae2 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.634474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7eefe5-6f6e-47ac-8622-02b2edb38ae2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.634474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7eefe5-6f6e-47ac-8622-02b2edb38ae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.652283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2a7eefe5-6f6e-47ac-8622-02b2edb38ae2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:11.653471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2a7eefe5-6f6e-47ac-8622-02b2edb38ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:11.672405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:11.673022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.673332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.673580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.674263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:11.674566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:11.676024 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:11.678431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:11.684967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 32/64] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:11.685764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:11.721449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:11.743292 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 33/65] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:11.744061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7eefe5-6f6e-47ac-8622-02b2edb38ae2 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:11.806607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae078096-35b1-4c22-abe7-7f9a9ff29014 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:11.857107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4652c411-41ed-49b6-b9b7-fccc5b4d4e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:11.857788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4652c411-41ed-49b6-b9b7-fccc5b4d4e25 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.858008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4652c411-41ed-49b6-b9b7-fccc5b4d4e25 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.858532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4652c411-41ed-49b6-b9b7-fccc5b4d4e25 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.873407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4652c411-41ed-49b6-b9b7-fccc5b4d4e25 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:11.875624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7eefe5-6f6e-47ac-8622-02b2edb38ae2 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-4652c411-41ed-49b6-b9b7-fccc5b4d4e25 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:11.879443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:11.880287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:11.881035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 33/66] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:42:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:11.881644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4652c411-41ed-49b6-b9b7-fccc5b4d4e25 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:11.916072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:11.916694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:11.916978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:11.917286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:11.917955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] RBAC: Adding query filter params (name=member) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:11.918266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:11.919687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:11.928748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 34/67] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:11.929535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:11.964038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:11.986915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:11.997282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4652c411-41ed-49b6-b9b7-fccc5b4d4e25 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.032651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff79345-d84a-43db-9966-5c64a94b7fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.033273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff79345-d84a-43db-9966-5c64a94b7fd4 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.033554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff79345-d84a-43db-9966-5c64a94b7fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.033825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff79345-d84a-43db-9966-5c64a94b7fd4 None admin] PATH_INFO: `/v3/users/4cc834c20ef040c19350af5fc47f4735` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.034978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff79345-d84a-43db-9966-5c64a94b7fd4 None admin] RBAC: Authorizing `identity:get_user(user_id=4cc834c20ef040c19350af5fc47f4735)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.036437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff79345-d84a-43db-9966-5c64a94b7fd4 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.038005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 34/68] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:11 2022] GET /identity/v3/users/4cc834c20ef040c19350af5fc47f4735 => generated 247 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:12.040139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8cfe5-71ea-4e88-8e94-d14f0284c41b None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.040431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8cfe5-71ea-4e88-8e94-d14f0284c41b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.040712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc8cfe5-71ea-4e88-8e94-d14f0284c41b None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.042193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 35/69] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:12.042799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc8cfe5-71ea-4e88-8e94-d14f0284c41b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.048228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd6d697-a0d7-484b-bc4a-0535f7dd9f0a None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.084398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.085003 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.085283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.085556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.086216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:12.086621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.088694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.097336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 35/70] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:42:11 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:12.097336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.153383 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc8cfe5-71ea-4e88-8e94-d14f0284c41b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.180865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:12.186447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53b2023-4915-439a-8f2b-0b06c2a0789c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.187113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53b2023-4915-439a-8f2b-0b06c2a0789c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.187397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53b2023-4915-439a-8f2b-0b06c2a0789c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.187674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53b2023-4915-439a-8f2b-0b06c2a0789c None admin] PATH_INFO: `/v3/users/7a93405c40b34cf8989eb8d1076eb475` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.195791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80302491-df0d-499a-8c36-617d0ae7b649 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.200708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53b2023-4915-439a-8f2b-0b06c2a0789c None admin] RBAC: Authorizing `identity:get_user(user_id=7a93405c40b34cf8989eb8d1076eb475)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.202268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53b2023-4915-439a-8f2b-0b06c2a0789c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.203928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 36/71] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/users/7a93405c40b34cf8989eb8d1076eb475 => generated 275 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:12.204692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53b2023-4915-439a-8f2b-0b06c2a0789c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.224332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:12.234319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1fa061-c0d7-4782-b953-bc39da70c01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.236140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1fa061-c0d7-4782-b953-bc39da70c01c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.236140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1fa061-c0d7-4782-b953-bc39da70c01c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.236140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1fa061-c0d7-4782-b953-bc39da70c01c None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.246054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1fa061-c0d7-4782-b953-bc39da70c01c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-fb1fa061-c0d7-4782-b953-bc39da70c01c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:12.250874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:12.251790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 36/72] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:12.253116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355baf01-9e9b-406d-a5fd-47f5be8f46b5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.253998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355baf01-9e9b-406d-a5fd-47f5be8f46b5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.253998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355baf01-9e9b-406d-a5fd-47f5be8f46b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.272102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-355baf01-9e9b-406d-a5fd-47f5be8f46b5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:12.273305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-355baf01-9e9b-406d-a5fd-47f5be8f46b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:12.306411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53b2023-4915-439a-8f2b-0b06c2a0789c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.338245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 37/73] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:12.339688 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355baf01-9e9b-406d-a5fd-47f5be8f46b5 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.339806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb4d940-9825-48f5-a96c-4a2e69eca258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.340338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4d940-9825-48f5-a96c-4a2e69eca258 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.340620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4d940-9825-48f5-a96c-4a2e69eca258 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.340897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4d940-9825-48f5-a96c-4a2e69eca258 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.342053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb4d940-9825-48f5-a96c-4a2e69eca258 None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.343551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb4d940-9825-48f5-a96c-4a2e69eca258 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.345168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 37/74] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:12.345917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb4d940-9825-48f5-a96c-4a2e69eca258 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.433357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:12.434433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:12.447546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb4d940-9825-48f5-a96c-4a2e69eca258 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.461269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355baf01-9e9b-406d-a5fd-47f5be8f46b5 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.482565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03c496-fadd-488d-878e-40bc34605de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.483219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03c496-fadd-488d-878e-40bc34605de7 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.483506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03c496-fadd-488d-878e-40bc34605de7 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.483780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03c496-fadd-488d-878e-40bc34605de7 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.484897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c03c496-fadd-488d-878e-40bc34605de7 None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.486343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c03c496-fadd-488d-878e-40bc34605de7 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.487997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 38/75] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:12.488752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03c496-fadd-488d-878e-40bc34605de7 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.501805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f595ffa7-1688-430a-8ce2-3165288e70ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.502383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f595ffa7-1688-430a-8ce2-3165288e70ca None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.502665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f595ffa7-1688-430a-8ce2-3165288e70ca None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.502983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f595ffa7-1688-430a-8ce2-3165288e70ca None admin] PATH_INFO: `/v3/users/7a93405c40b34cf8989eb8d1076eb475` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.504389 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f595ffa7-1688-430a-8ce2-3165288e70ca None admin] RBAC: Authorizing `identity:get_user(user_id=7a93405c40b34cf8989eb8d1076eb475)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.505888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f595ffa7-1688-430a-8ce2-3165288e70ca None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.511996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 38/76] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/users/7a93405c40b34cf8989eb8d1076eb475 => generated 275 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:12.511996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f595ffa7-1688-430a-8ce2-3165288e70ca None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.601305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03c496-fadd-488d-878e-40bc34605de7 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.613793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f595ffa7-1688-430a-8ce2-3165288e70ca None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.635955 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.636568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.636850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.637128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.637799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] RBAC: Adding query filter params (name=member) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:12.638109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.639557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.646364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f94887b-2e31-4a6a-afda-ebc1c3537f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.647049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94887b-2e31-4a6a-afda-ebc1c3537f15 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.647456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 39/77] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:12.647554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94887b-2e31-4a6a-afda-ebc1c3537f15 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.647669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f94887b-2e31-4a6a-afda-ebc1c3537f15 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df/users/4cc834c20ef040c19350af5fc47f4735/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.648052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.649677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f94887b-2e31-4a6a-afda-ebc1c3537f15 None admin] RBAC: Authorizing `identity:create_grant(project_id=ebd577f63e0b41edb25465e82d9af4df, user_id=4cc834c20ef040c19350af5fc47f4735, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.651423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f94887b-2e31-4a6a-afda-ebc1c3537f15 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.668091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:12.713284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 39/78] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:12 2022] PUT /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df/users/4cc834c20ef040c19350af5fc47f4735/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:12.714702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f94887b-2e31-4a6a-afda-ebc1c3537f15 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.723170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:12.759804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e247994-22ab-49c7-b505-af5e2a5da09a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.805014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea19762-6b0a-4f39-bddd-8e20458ac683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.805696 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea19762-6b0a-4f39-bddd-8e20458ac683 None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.805978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea19762-6b0a-4f39-bddd-8e20458ac683 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.806259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea19762-6b0a-4f39-bddd-8e20458ac683 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df/users/7a93405c40b34cf8989eb8d1076eb475/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.808989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea19762-6b0a-4f39-bddd-8e20458ac683 None admin] RBAC: Authorizing `identity:create_grant(project_id=ebd577f63e0b41edb25465e82d9af4df, user_id=7a93405c40b34cf8989eb8d1076eb475, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.810484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea19762-6b0a-4f39-bddd-8e20458ac683 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.842390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f94887b-2e31-4a6a-afda-ebc1c3537f15 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:12.878028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8158f8db-3e65-4e78-b32b-76f41420813e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:12.878684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8158f8db-3e65-4e78-b32b-76f41420813e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:12.878934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8158f8db-3e65-4e78-b32b-76f41420813e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:12.879236 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8158f8db-3e65-4e78-b32b-76f41420813e None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:12.880371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8158f8db-3e65-4e78-b32b-76f41420813e None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:12.881882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8158f8db-3e65-4e78-b32b-76f41420813e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:12.883537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 40/79] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:12.884317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8158f8db-3e65-4e78-b32b-76f41420813e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.904669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 40/80] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:12 2022] PUT /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df/users/7a93405c40b34cf8989eb8d1076eb475/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:12.911106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea19762-6b0a-4f39-bddd-8e20458ac683 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:12.964272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:13.000840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8158f8db-3e65-4e78-b32b-76f41420813e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:13.023226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:13.040215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d0be5-87fe-4e96-b022-50f1e9d883d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:13.040858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d0be5-87fe-4e96-b022-50f1e9d883d9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:13.041141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d0be5-87fe-4e96-b022-50f1e9d883d9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:13.041413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82d0be5-87fe-4e96-b022-50f1e9d883d9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:13.043121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82d0be5-87fe-4e96-b022-50f1e9d883d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:13.044602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82d0be5-87fe-4e96-b022-50f1e9d883d9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:13.046550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 41/81] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:13.047041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d0be5-87fe-4e96-b022-50f1e9d883d9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:13.064949 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea19762-6b0a-4f39-bddd-8e20458ac683 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:13.107715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2a110f-800d-4c56-8737-62bd42197cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:13.108862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2a110f-800d-4c56-8737-62bd42197cdc None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:13.109212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2a110f-800d-4c56-8737-62bd42197cdc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:13.109560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf2a110f-800d-4c56-8737-62bd42197cdc None admin] PATH_INFO: `/v3/users/7a93405c40b34cf8989eb8d1076eb475` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:13.111216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2a110f-800d-4c56-8737-62bd42197cdc None admin] RBAC: Authorizing `identity:get_user(user_id=7a93405c40b34cf8989eb8d1076eb475)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:13.113058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf2a110f-800d-4c56-8737-62bd42197cdc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:13.114239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 41/82] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:12 2022] GET /identity/v3/users/7a93405c40b34cf8989eb8d1076eb475 => generated 275 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:13.114760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2a110f-800d-4c56-8737-62bd42197cdc None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:13.173887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82d0be5-87fe-4e96-b022-50f1e9d883d9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:13.213623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d1c9b2-a17b-4542-b5bc-f3bdf54d32b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:13.215030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d1c9b2-a17b-4542-b5bc-f3bdf54d32b6 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:13.215364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d1c9b2-a17b-4542-b5bc-f3bdf54d32b6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:13.215649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d1c9b2-a17b-4542-b5bc-f3bdf54d32b6 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df/users/7a93405c40b34cf8989eb8d1076eb475/roles/3cb31ad012324a57acd3a4bd258d1fb1` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:13.219806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d1c9b2-a17b-4542-b5bc-f3bdf54d32b6 None admin] RBAC: Authorizing `identity:create_grant(project_id=ebd577f63e0b41edb25465e82d9af4df, user_id=7a93405c40b34cf8989eb8d1076eb475, role_id=3cb31ad012324a57acd3a4bd258d1fb1)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:13.221562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d1c9b2-a17b-4542-b5bc-f3bdf54d32b6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:13.240903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf2a110f-800d-4c56-8737-62bd42197cdc None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:13.276314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bdccd0-f7bd-4aa6-a684-65fc709acef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:13.277431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bdccd0-f7bd-4aa6-a684-65fc709acef9 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:13.277714 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bdccd0-f7bd-4aa6-a684-65fc709acef9 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:13.277987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bdccd0-f7bd-4aa6-a684-65fc709acef9 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:13.278445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:13.278646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bdccd0-f7bd-4aa6-a684-65fc709acef9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:13.280139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bdccd0-f7bd-4aa6-a684-65fc709acef9 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:13.282439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 42/83] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:13 2022] PUT /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df/users/7a93405c40b34cf8989eb8d1076eb475/roles/3cb31ad012324a57acd3a4bd258d1fb1 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:13.286646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d1c9b2-a17b-4542-b5bc-f3bdf54d32b6 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:13.393841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:13.409684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 42/84] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:13 2022] POST /identity/v3/users => generated 289 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:13.416106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d1c9b2-a17b-4542-b5bc-f3bdf54d32b6 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:13.461440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea4e607f-48ed-4f19-9584-8fba6c5e8e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:13.462160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4e607f-48ed-4f19-9584-8fba6c5e8e36 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:13.462493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4e607f-48ed-4f19-9584-8fba6c5e8e36 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:13.462776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea4e607f-48ed-4f19-9584-8fba6c5e8e36 None admin] PATH_INFO: `/v3/projects/b108a7dedac34f2297cb274ca4f5f599` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:13.464809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4e607f-48ed-4f19-9584-8fba6c5e8e36 None admin] RBAC: Authorizing `identity:get_project(project_id=b108a7dedac34f2297cb274ca4f5f599)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:13.466516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea4e607f-48ed-4f19-9584-8fba6c5e8e36 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:13.468799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 43/85] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:13 2022] GET /identity/v3/projects/b108a7dedac34f2297cb274ca4f5f599 => generated 312 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:13.476142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bdccd0-f7bd-4aa6-a684-65fc709acef9 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:13.597520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bdccd0-f7bd-4aa6-a684-65fc709acef9 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:13.639260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9727c9ce-efac-41c2-83c4-a4c15bb6825c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:13.639965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727c9ce-efac-41c2-83c4-a4c15bb6825c None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:13.640250 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727c9ce-efac-41c2-83c4-a4c15bb6825c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:13.640526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9727c9ce-efac-41c2-83c4-a4c15bb6825c None admin] PATH_INFO: `/v3/projects/b108a7dedac34f2297cb274ca4f5f599/users/7a93405c40b34cf8989eb8d1076eb475/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:13.644360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9727c9ce-efac-41c2-83c4-a4c15bb6825c None admin] RBAC: Authorizing `identity:create_grant(project_id=b108a7dedac34f2297cb274ca4f5f599, user_id=7a93405c40b34cf8989eb8d1076eb475, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:13.645958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9727c9ce-efac-41c2-83c4-a4c15bb6825c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:13.715372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 43/86] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:13 2022] PUT /identity/v3/projects/b108a7dedac34f2297cb274ca4f5f599/users/7a93405c40b34cf8989eb8d1076eb475/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:15.173469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3cf14-43df-4cc8-8a83-3d33b9237dff None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:15.173957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3cf14-43df-4cc8-8a83-3d33b9237dff None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:15.173957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e3cf14-43df-4cc8-8a83-3d33b9237dff None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:15.175567 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 44/87] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:15.191579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404cd76c-df73-426a-8fc0-8b2952f21002 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:15.192049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404cd76c-df73-426a-8fc0-8b2952f21002 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:15.192119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404cd76c-df73-426a-8fc0-8b2952f21002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:15.216848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-404cd76c-df73-426a-8fc0-8b2952f21002 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:15.218060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-404cd76c-df73-426a-8fc0-8b2952f21002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:15.290377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:15.303946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 44/88] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:15.472256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ed5c2a-07d5-4ae4-b802-9009e85c4bc2 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:15.472585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ed5c2a-07d5-4ae4-b802-9009e85c4bc2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:15.472814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78ed5c2a-07d5-4ae4-b802-9009e85c4bc2 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:15.474162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 45/89] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:15.479613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8851d4-180f-403d-857e-f5f0500871fb None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:15.479912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8851d4-180f-403d-857e-f5f0500871fb None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:15.480191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8851d4-180f-403d-857e-f5f0500871fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:15.504557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6a8851d4-180f-403d-857e-f5f0500871fb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:15.505736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6a8851d4-180f-403d-857e-f5f0500871fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:15.586041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 45/90] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:15.764606 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ed5c2a-07d5-4ae4-b802-9009e85c4bc2 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:15.854890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:15.902144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78ed5c2a-07d5-4ae4-b802-9009e85c4bc2 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:15.947687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3eec80-86b5-45ae-999f-b421f286fe0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:15.948856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3eec80-86b5-45ae-999f-b421f286fe0c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:15.949149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3eec80-86b5-45ae-999f-b421f286fe0c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:15.949422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3eec80-86b5-45ae-999f-b421f286fe0c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:15.963921 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3eec80-86b5-45ae-999f-b421f286fe0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-de3eec80-86b5-45ae-999f-b421f286fe0c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:15.969738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:15.970580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:15.971366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 46/91] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:42:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:15.980654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8851d4-180f-403d-857e-f5f0500871fb None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:16.079035 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3eec80-86b5-45ae-999f-b421f286fe0c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:16.114837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8851d4-180f-403d-857e-f5f0500871fb None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:16.141393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:16.158393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:16.159285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:16.159569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:16.160316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:16.161156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:16.161480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:16.162981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:16.172702 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 46/92] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:16.179059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:16.306176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3eec80-86b5-45ae-999f-b421f286fe0c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:16.342211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f1a1b77-b8f3-43aa-8288-6a1f3a142d2f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:16.349447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:16.358862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1061902-62c1-4fe5-897e-191e6a3e08a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:16.359682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1061902-62c1-4fe5-897e-191e6a3e08a6 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:16.359966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1061902-62c1-4fe5-897e-191e6a3e08a6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:16.360238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1061902-62c1-4fe5-897e-191e6a3e08a6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:16.371063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1061902-62c1-4fe5-897e-191e6a3e08a6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-f1061902-62c1-4fe5-897e-191e6a3e08a6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:16.375954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:16.376796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:16.377554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 47/93] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:16.378645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf28a042-5bff-4a99-9fe8-557bad059d44 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:16.378645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf28a042-5bff-4a99-9fe8-557bad059d44 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:16.378984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf28a042-5bff-4a99-9fe8-557bad059d44 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:16.380342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 48/94] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:16.389442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624524a-d886-4423-b8dc-978fc1f8b7e1 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:16.389732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624524a-d886-4423-b8dc-978fc1f8b7e1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:16.390010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9624524a-d886-4423-b8dc-978fc1f8b7e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:16.396128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827a868-12a2-4bb3-bcfb-9f988642a0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:16.396767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827a868-12a2-4bb3-bcfb-9f988642a0fb None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:16.397050 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827a868-12a2-4bb3-bcfb-9f988642a0fb None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:16.397326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827a868-12a2-4bb3-bcfb-9f988642a0fb None admin] PATH_INFO: `/v3/users/4cc834c20ef040c19350af5fc47f4735` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:16.398475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5827a868-12a2-4bb3-bcfb-9f988642a0fb None admin] RBAC: Authorizing `identity:get_user(user_id=4cc834c20ef040c19350af5fc47f4735)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:16.400198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5827a868-12a2-4bb3-bcfb-9f988642a0fb None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:16.402870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 47/95] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:16 2022] GET /identity/v3/users/4cc834c20ef040c19350af5fc47f4735 => generated 247 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:16.403588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827a868-12a2-4bb3-bcfb-9f988642a0fb None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:16.425826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-9624524a-d886-4423-b8dc-978fc1f8b7e1 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:16.427061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9624524a-d886-4423-b8dc-978fc1f8b7e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:16.491331 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:16.524922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 49/96] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:16.526052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624524a-d886-4423-b8dc-978fc1f8b7e1 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:16.547464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827a868-12a2-4bb3-bcfb-9f988642a0fb None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:16.615153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:16.615741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:16.616026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:16.616300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:16.617029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] RBAC: Adding query filter params (name=member) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:16.617337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:16.618889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:16.628587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 48/97] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:16.634312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:16.706670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9624524a-d886-4423-b8dc-978fc1f8b7e1 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:16.724051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:16.747748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c23cc4-ddd8-42e2-9a8d-1be73527df95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:16.748454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c23cc4-ddd8-42e2-9a8d-1be73527df95 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:16.748799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c23cc4-ddd8-42e2-9a8d-1be73527df95 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:16.749141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c23cc4-ddd8-42e2-9a8d-1be73527df95 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:16.816342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be651bb-c0cd-4d6c-be23-ca3fecaeef37 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:16.821449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c23cc4-ddd8-42e2-9a8d-1be73527df95 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:16.823155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c23cc4-ddd8-42e2-9a8d-1be73527df95 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:16.845579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:16.911767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7714f91-bc03-4667-a0cc-b7bc00ace758 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:16.912332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7714f91-bc03-4667-a0cc-b7bc00ace758 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:16.912619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7714f91-bc03-4667-a0cc-b7bc00ace758 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:16.912896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7714f91-bc03-4667-a0cc-b7bc00ace758 None admin] PATH_INFO: `/v3/users/7a93405c40b34cf8989eb8d1076eb475` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:16.914100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7714f91-bc03-4667-a0cc-b7bc00ace758 None admin] RBAC: Authorizing `identity:get_user(user_id=7a93405c40b34cf8989eb8d1076eb475)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:16.915605 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7714f91-bc03-4667-a0cc-b7bc00ace758 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:16.917836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 49/98] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:16 2022] GET /identity/v3/users/7a93405c40b34cf8989eb8d1076eb475 => generated 275 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:16.924763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7714f91-bc03-4667-a0cc-b7bc00ace758 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:16.989542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 50/99] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 464 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:16.990515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c23cc4-ddd8-42e2-9a8d-1be73527df95 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:17.024148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:17.145355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7714f91-bc03-4667-a0cc-b7bc00ace758 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:17.196891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c23cc4-ddd8-42e2-9a8d-1be73527df95 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:17.200891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf905173-9255-4ea8-8aeb-c6647e2c87c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:17.201547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf905173-9255-4ea8-8aeb-c6647e2c87c2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.201831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf905173-9255-4ea8-8aeb-c6647e2c87c2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.202106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf905173-9255-4ea8-8aeb-c6647e2c87c2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.248788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e97541-9568-4005-8a83-99214f5e4057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:17.249425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e97541-9568-4005-8a83-99214f5e4057 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.249705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e97541-9568-4005-8a83-99214f5e4057 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.249980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e97541-9568-4005-8a83-99214f5e4057 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.260424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e97541-9568-4005-8a83-99214f5e4057 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:17.265651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-49e97541-9568-4005-8a83-99214f5e4057 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:17.265767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:17.266721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 51/100] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:42:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:17.268063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e97541-9568-4005-8a83-99214f5e4057 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:17.293098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf905173-9255-4ea8-8aeb-c6647e2c87c2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:17.294602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf905173-9255-4ea8-8aeb-c6647e2c87c2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:17.345299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:17.368122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e97541-9568-4005-8a83-99214f5e4057 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:17.392951 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 50/101] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:17.394204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0e1e21-e47b-48e6-8e69-930557095307 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.394494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0e1e21-e47b-48e6-8e69-930557095307 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.394772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0e1e21-e47b-48e6-8e69-930557095307 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.396053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 51/102] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:17.397062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3539230-59c1-4475-a38c-1b533f9aac9f None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.397337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3539230-59c1-4475-a38c-1b533f9aac9f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.397666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3539230-59c1-4475-a38c-1b533f9aac9f None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.398831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 52/103] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:17.399557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3539230-59c1-4475-a38c-1b533f9aac9f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:17.406266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f7ad1-d07f-4c41-b825-343e4fbb8dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:17.406900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f7ad1-d07f-4c41-b825-343e4fbb8dfd None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.407209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f7ad1-d07f-4c41-b825-343e4fbb8dfd None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.407482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f7ad1-d07f-4c41-b825-343e4fbb8dfd None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.408603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152f7ad1-d07f-4c41-b825-343e4fbb8dfd None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:17.410061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152f7ad1-d07f-4c41-b825-343e4fbb8dfd None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:17.411702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 52/104] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:17 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:17.412813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1564c-0595-49a1-93e2-9973bc84cb20 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.413089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1564c-0595-49a1-93e2-9973bc84cb20 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.413360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e1564c-0595-49a1-93e2-9973bc84cb20 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.414533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 53/105] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:17.419468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e1564c-0595-49a1-93e2-9973bc84cb20 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:17.487102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:17.506522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3539230-59c1-4475-a38c-1b533f9aac9f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:17.519467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e1564c-0595-49a1-93e2-9973bc84cb20 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:17.540160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4b3ad-6228-487f-95db-0cd310e40ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:17.540759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4b3ad-6228-487f-95db-0cd310e40ed0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.541042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4b3ad-6228-487f-95db-0cd310e40ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.541322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4b3ad-6228-487f-95db-0cd310e40ed0 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.541988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4b3ad-6228-487f-95db-0cd310e40ed0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:17.542365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4b3ad-6228-487f-95db-0cd310e40ed0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:17.543793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4b3ad-6228-487f-95db-0cd310e40ed0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:17.551358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 53/106] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:42:17 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:17.552562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd9da04-84f3-41f2-aafd-f6dcf23736d3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.552693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc7de08-7f48-474b-a2d9-2a9f60c7d320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:17.552865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd9da04-84f3-41f2-aafd-f6dcf23736d3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.553141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd9da04-84f3-41f2-aafd-f6dcf23736d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.553228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7de08-7f48-474b-a2d9-2a9f60c7d320 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.553469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7de08-7f48-474b-a2d9-2a9f60c7d320 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.553745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc7de08-7f48-474b-a2d9-2a9f60c7d320 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.554857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc7de08-7f48-474b-a2d9-2a9f60c7d320 None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:17.556315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc7de08-7f48-474b-a2d9-2a9f60c7d320 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:17.558440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 54/107] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:17 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:17.559526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df10db1-5453-4aab-9c5d-4da40406ab1c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.559809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df10db1-5453-4aab-9c5d-4da40406ab1c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.560078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df10db1-5453-4aab-9c5d-4da40406ab1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.575338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1fd9da04-84f3-41f2-aafd-f6dcf23736d3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:17.576522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1fd9da04-84f3-41f2-aafd-f6dcf23736d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:17.577450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-4df10db1-5453-4aab-9c5d-4da40406ab1c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:17.578601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4df10db1-5453-4aab-9c5d-4da40406ab1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:17.585784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:17.641654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 54/108] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:17.642488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd9da04-84f3-41f2-aafd-f6dcf23736d3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:17.656456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 55/109] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:17.656456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6534f-3ff4-4345-a1ef-a627bf3bf2f7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.656456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6534f-3ff4-4345-a1ef-a627bf3bf2f7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.656456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6534f-3ff4-4345-a1ef-a627bf3bf2f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.676342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6ff6534f-3ff4-4345-a1ef-a627bf3bf2f7 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:17.677528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6ff6534f-3ff4-4345-a1ef-a627bf3bf2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:17.740000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:17.747661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 56/110] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:17.755334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6534f-3ff4-4345-a1ef-a627bf3bf2f7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:17.776632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd9da04-84f3-41f2-aafd-f6dcf23736d3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:17.858283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:17.858910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.859219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.859492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.860230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=ebd577f63e0b41edb25465e82d9af4df, user.id=4cc834c20ef040c19350af5fc47f4735) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:17.860536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:17.861986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:17.878055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 55/111] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:17 2022] GET /identity/v3/role_assignments?user.id=4cc834c20ef040c19350af5fc47f4735&scope.project.id=ebd577f63e0b41edb25465e82d9af4df&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 608 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:17.878861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:17.883659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:17.923624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6534f-3ff4-4345-a1ef-a627bf3bf2f7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:17.968570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7469c881-09c0-49e7-9e8a-a80645b3e7f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:17.970105 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469c881-09c0-49e7-9e8a-a80645b3e7f8 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:17.970536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469c881-09c0-49e7-9e8a-a80645b3e7f8 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:17.971099 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7469c881-09c0-49e7-9e8a-a80645b3e7f8 None admin] PATH_INFO: `/v3/users/7a93405c40b34cf8989eb8d1076eb475` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:17.972554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7469c881-09c0-49e7-9e8a-a80645b3e7f8 None admin] RBAC: Authorizing `identity:get_user(user_id=7a93405c40b34cf8989eb8d1076eb475)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:17.974250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7469c881-09c0-49e7-9e8a-a80645b3e7f8 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:17.977188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 57/112] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:17 2022] GET /identity/v3/users/7a93405c40b34cf8989eb8d1076eb475 => generated 275 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:17.978480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7469c881-09c0-49e7-9e8a-a80645b3e7f8 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.014289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb551fe4-aebe-42cc-a641-7e697373db4a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.054013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.054642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.054929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.055251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.056010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] RBAC: Adding query filter params (role.id=974299ae4a414939a4f7e186921d4569, scope.project.id=ebd577f63e0b41edb25465e82d9af4df, user.id=7a93405c40b34cf8989eb8d1076eb475) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:18.056324 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:18.058376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:18.061678 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:18.065273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 56/113] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:17 2022] GET /identity/v3/role_assignments?user.id=7a93405c40b34cf8989eb8d1076eb475&scope.project.id=ebd577f63e0b41edb25465e82d9af4df&role.id=974299ae4a414939a4f7e186921d4569 => generated 608 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:18.067179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.115254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7469c881-09c0-49e7-9e8a-a80645b3e7f8 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.164446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708a0cf-f6f2-4386-b32c-4752febdafd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.165098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708a0cf-f6f2-4386-b32c-4752febdafd5 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.165379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708a0cf-f6f2-4386-b32c-4752febdafd5 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.165667 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708a0cf-f6f2-4386-b32c-4752febdafd5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.166832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e708a0cf-f6f2-4386-b32c-4752febdafd5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:18.169438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e708a0cf-f6f2-4386-b32c-4752febdafd5 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:18.171607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 58/114] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:18.172216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708a0cf-f6f2-4386-b32c-4752febdafd5 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.185740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:18.227368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a81c0df-3c10-4580-9a8f-77ada16f644c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.272865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87effd07-d885-494d-9166-da9caee80314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.273414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87effd07-d885-494d-9166-da9caee80314 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.273702 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87effd07-d885-494d-9166-da9caee80314 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.273982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87effd07-d885-494d-9166-da9caee80314 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.303696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708a0cf-f6f2-4386-b32c-4752febdafd5 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.337677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2ee13c-e8c0-491b-aa71-c7a07a900393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.338309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ee13c-e8c0-491b-aa71-c7a07a900393 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.338589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ee13c-e8c0-491b-aa71-c7a07a900393 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.338869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2ee13c-e8c0-491b-aa71-c7a07a900393 None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.346588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:18.348841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87effd07-d885-494d-9166-da9caee80314 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:18.349480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2ee13c-e8c0-491b-aa71-c7a07a900393 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:18.350335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87effd07-d885-494d-9166-da9caee80314 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-3b2ee13c-e8c0-491b-aa71-c7a07a900393 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:18.354268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:18.355754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:18.356495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 59/115] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:18.357108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2ee13c-e8c0-491b-aa71-c7a07a900393 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.422616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:18.447829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 57/116] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:18.447829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87effd07-d885-494d-9166-da9caee80314 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.455366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2ee13c-e8c0-491b-aa71-c7a07a900393 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.488807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1639bce-e1e2-4876-abdc-80c1de1957ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.489432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1639bce-e1e2-4876-abdc-80c1de1957ae None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.489715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1639bce-e1e2-4876-abdc-80c1de1957ae None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.489991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1639bce-e1e2-4876-abdc-80c1de1957ae None admin] PATH_INFO: `/v3/roles/reader` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.500649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1639bce-e1e2-4876-abdc-80c1de1957ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-f1639bce-e1e2-4876-abdc-80c1de1957ae None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:18.505455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:18.506308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:18.507094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 60/117] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:18.507796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1639bce-e1e2-4876-abdc-80c1de1957ae None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.547757 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:18.550596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87effd07-d885-494d-9166-da9caee80314 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.582878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56014ba1-c968-4618-953d-8d784618a486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.583739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56014ba1-c968-4618-953d-8d784618a486 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.584033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56014ba1-c968-4618-953d-8d784618a486 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.584521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56014ba1-c968-4618-953d-8d784618a486 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.585597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56014ba1-c968-4618-953d-8d784618a486 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:18.587122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56014ba1-c968-4618-953d-8d784618a486 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:18.609525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1639bce-e1e2-4876-abdc-80c1de1957ae None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.635030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:18.643134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.643785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.644065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.644338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.645005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] RBAC: Adding query filter params (name=member) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:18.645310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:18.646703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:18.655296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 61/118] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:18.656120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.664063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 58/119] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:42:18 2022] POST /identity/v3/projects => generated 302 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:18.664938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56014ba1-c968-4618-953d-8d784618a486 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.770837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53971c55-acba-445e-8793-38e5092c07c6 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.777873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56014ba1-c968-4618-953d-8d784618a486 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.812179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e8fda3-a701-40f1-879b-06de0112a46c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.812745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e8fda3-a701-40f1-879b-06de0112a46c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.813032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e8fda3-a701-40f1-879b-06de0112a46c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.813306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6e8fda3-a701-40f1-879b-06de0112a46c None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.814482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e8fda3-a701-40f1-879b-06de0112a46c None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:18.815987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6e8fda3-a701-40f1-879b-06de0112a46c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:18.822807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 62/120] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:18.822807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e8fda3-a701-40f1-879b-06de0112a46c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.831188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.831188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.831453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.831715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.832399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] RBAC: Adding query filter params (name=reader) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:18.832713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:18.834113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:18.850691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 59/121] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:18.850691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:18.867194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:18.891819 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:18.936687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6e8fda3-a701-40f1-879b-06de0112a46c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.958598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5aacd08-c2f7-4125-b27f-834d52073de3 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:18.970087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6e5dba-aa28-4c9c-869c-5baf6cc510ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:18.970720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e5dba-aa28-4c9c-869c-5baf6cc510ee None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:18.971027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e5dba-aa28-4c9c-869c-5baf6cc510ee None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:18.971308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6e5dba-aa28-4c9c-869c-5baf6cc510ee None admin] PATH_INFO: `/v3/users/7a93405c40b34cf8989eb8d1076eb475` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:18.972457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6e5dba-aa28-4c9c-869c-5baf6cc510ee None admin] RBAC: Authorizing `identity:get_user(user_id=7a93405c40b34cf8989eb8d1076eb475)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:18.973925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6e5dba-aa28-4c9c-869c-5baf6cc510ee None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:18.975542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 63/122] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/users/7a93405c40b34cf8989eb8d1076eb475 => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:18.976307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6e5dba-aa28-4c9c-869c-5baf6cc510ee None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:19.004567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:19.005490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:19.005771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:19.006049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:19.006784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] RBAC: Adding query filter params (role.id=3cb31ad012324a57acd3a4bd258d1fb1, scope.project.id=ebd577f63e0b41edb25465e82d9af4df, user.id=7a93405c40b34cf8989eb8d1076eb475) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:19.007125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:19.008578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:19.015588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 60/123] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/role_assignments?user.id=7a93405c40b34cf8989eb8d1076eb475&scope.project.id=ebd577f63e0b41edb25465e82d9af4df&role.id=3cb31ad012324a57acd3a4bd258d1fb1 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:19.016193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:19.099745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6e5dba-aa28-4c9c-869c-5baf6cc510ee None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:19.120779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:19.128195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:19.138542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1db2a8-08b2-4d9d-8f7e-1cb9c5322035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:19.139257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1db2a8-08b2-4d9d-8f7e-1cb9c5322035 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:19.139524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1db2a8-08b2-4d9d-8f7e-1cb9c5322035 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:19.139807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1db2a8-08b2-4d9d-8f7e-1cb9c5322035 None admin] PATH_INFO: `/v3/users/06564f22f17a4b3697b3d27f62ee1e7d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:19.148052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c107c0d4-18a4-4310-936d-6ae8b596a740 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:19.158149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1db2a8-08b2-4d9d-8f7e-1cb9c5322035 None admin] RBAC: Authorizing `identity:get_user(user_id=06564f22f17a4b3697b3d27f62ee1e7d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:19.159839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1db2a8-08b2-4d9d-8f7e-1cb9c5322035 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:19.161827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 64/124] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:18 2022] GET /identity/v3/users/06564f22f17a4b3697b3d27f62ee1e7d => generated 289 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:19.168197 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1db2a8-08b2-4d9d-8f7e-1cb9c5322035 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:19.190797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbb9a7f-0eab-4119-8094-00412468e699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:19.190797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbb9a7f-0eab-4119-8094-00412468e699 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:19.190797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbb9a7f-0eab-4119-8094-00412468e699 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:19.191130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bbb9a7f-0eab-4119-8094-00412468e699 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:19.278434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbb9a7f-0eab-4119-8094-00412468e699 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:19.280012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bbb9a7f-0eab-4119-8094-00412468e699 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:19.287397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1db2a8-08b2-4d9d-8f7e-1cb9c5322035 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:19.326785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f20a921-d595-45cc-8976-c8ceea03a22d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:19.327605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f20a921-d595-45cc-8976-c8ceea03a22d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:19.328268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f20a921-d595-45cc-8976-c8ceea03a22d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:19.328553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f20a921-d595-45cc-8976-c8ceea03a22d None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:19.329844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f20a921-d595-45cc-8976-c8ceea03a22d None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:19.331354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f20a921-d595-45cc-8976-c8ceea03a22d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:19.333210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 65/125] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:19 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:19.338534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f20a921-d595-45cc-8976-c8ceea03a22d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:19.364176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:19.386548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 61/126] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:19.392118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbb9a7f-0eab-4119-8094-00412468e699 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:19.414148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:19.465982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f20a921-d595-45cc-8976-c8ceea03a22d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:19.491214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bbb9a7f-0eab-4119-8094-00412468e699 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:19.505475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9171804-250b-494e-b8c2-e11f9b5dad51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:19.506140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9171804-250b-494e-b8c2-e11f9b5dad51 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:19.506421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9171804-250b-494e-b8c2-e11f9b5dad51 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:19.506698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9171804-250b-494e-b8c2-e11f9b5dad51 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df/users/06564f22f17a4b3697b3d27f62ee1e7d/roles/2d48ee5817eb4e4388e1a55c7db0b40d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:19.508713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9171804-250b-494e-b8c2-e11f9b5dad51 None admin] RBAC: Authorizing `identity:create_grant(project_id=ebd577f63e0b41edb25465e82d9af4df, user_id=06564f22f17a4b3697b3d27f62ee1e7d, role_id=2d48ee5817eb4e4388e1a55c7db0b40d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:19.510273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9171804-250b-494e-b8c2-e11f9b5dad51 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:19.525037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4c0dd-8b58-45cf-a9b4-94d78af6d4a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:19.525929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4c0dd-8b58-45cf-a9b4-94d78af6d4a1 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:19.526211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4c0dd-8b58-45cf-a9b4-94d78af6d4a1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:19.526513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc4c0dd-8b58-45cf-a9b4-94d78af6d4a1 None admin] PATH_INFO: `/v3/projects/b108a7dedac34f2297cb274ca4f5f599` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:19.527662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4c0dd-8b58-45cf-a9b4-94d78af6d4a1 None admin] RBAC: Authorizing `identity:get_project(project_id=b108a7dedac34f2297cb274ca4f5f599)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:19.529095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc4c0dd-8b58-45cf-a9b4-94d78af6d4a1 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:19.530721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 62/127] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:19 2022] GET /identity/v3/projects/b108a7dedac34f2297cb274ca4f5f599 => generated 312 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:19.536520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4c0dd-8b58-45cf-a9b4-94d78af6d4a1 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:19.580063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 66/128] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:19 2022] PUT /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df/users/06564f22f17a4b3697b3d27f62ee1e7d/roles/2d48ee5817eb4e4388e1a55c7db0b40d => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:19.587222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:19.649104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc4c0dd-8b58-45cf-a9b4-94d78af6d4a1 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:19.685842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1854d0b4-66e9-4854-b864-9774ce16700b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:19.686471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1854d0b4-66e9-4854-b864-9774ce16700b None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:19.686754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1854d0b4-66e9-4854-b864-9774ce16700b None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:19.687062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1854d0b4-66e9-4854-b864-9774ce16700b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:19.687820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1854d0b4-66e9-4854-b864-9774ce16700b None admin] RBAC: Adding query filter params (role.id=974299ae4a414939a4f7e186921d4569, scope.project.id=b108a7dedac34f2297cb274ca4f5f599, user.id=7a93405c40b34cf8989eb8d1076eb475) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:19.688137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1854d0b4-66e9-4854-b864-9774ce16700b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:19.689616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1854d0b4-66e9-4854-b864-9774ce16700b None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:19.702994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 63/129] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:19 2022] GET /identity/v3/role_assignments?user.id=7a93405c40b34cf8989eb8d1076eb475&scope.project.id=b108a7dedac34f2297cb274ca4f5f599&role.id=974299ae4a414939a4f7e186921d4569 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:20.864262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e7c2b3-7870-4ab3-9479-ac571b035430 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:20.864793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e7c2b3-7870-4ab3-9479-ac571b035430 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:20.864793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e7c2b3-7870-4ab3-9479-ac571b035430 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:20.866148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 67/130] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:20.871981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95cf9c-fab4-4bdd-9ad3-c278cfd07557 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:20.872389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95cf9c-fab4-4bdd-9ad3-c278cfd07557 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:20.872533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e95cf9c-fab4-4bdd-9ad3-c278cfd07557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:20.890741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-7e95cf9c-fab4-4bdd-9ad3-c278cfd07557 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:20.891958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7e95cf9c-fab4-4bdd-9ad3-c278cfd07557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:20.956520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 64/131] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:21.222727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e7c2b3-7870-4ab3-9479-ac571b035430 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:21.320224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e7c2b3-7870-4ab3-9479-ac571b035430 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:21.332981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:21.352892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a6d1ce-4771-47bf-bb85-9acf149fba72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:21.358468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a6d1ce-4771-47bf-bb85-9acf149fba72 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:21.358468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a6d1ce-4771-47bf-bb85-9acf149fba72 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:21.358468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a6d1ce-4771-47bf-bb85-9acf149fba72 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:21.358468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a6d1ce-4771-47bf-bb85-9acf149fba72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:21.358468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a6d1ce-4771-47bf-bb85-9acf149fba72 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:21.358807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 68/132] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:21.364314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e95cf9c-fab4-4bdd-9ad3-c278cfd07557 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:21.404568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:21.460241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e95cf9c-fab4-4bdd-9ad3-c278cfd07557 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:21.493164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120fbbc-73d0-432f-bb7d-583d8654a7a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:21.493711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120fbbc-73d0-432f-bb7d-583d8654a7a8 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:21.493991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120fbbc-73d0-432f-bb7d-583d8654a7a8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:21.494264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120fbbc-73d0-432f-bb7d-583d8654a7a8 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:21.494909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120fbbc-73d0-432f-bb7d-583d8654a7a8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:21.496423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120fbbc-73d0-432f-bb7d-583d8654a7a8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:21.560528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 65/133] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:21 2022] POST /identity/v3/users => generated 283 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:42:21.721838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e5eac2-38ba-437f-9fe5-5782dfeb7063 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:21.722127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e5eac2-38ba-437f-9fe5-5782dfeb7063 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:21.722409 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e5eac2-38ba-437f-9fe5-5782dfeb7063 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:21.723806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 69/134] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:21.729174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc61a25-fa91-4162-8554-013e05dfe4f4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:21.729460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc61a25-fa91-4162-8554-013e05dfe4f4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:21.730287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc61a25-fa91-4162-8554-013e05dfe4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:21.748958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-bfc61a25-fa91-4162-8554-013e05dfe4f4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:21.750343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-bfc61a25-fa91-4162-8554-013e05dfe4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:21.824704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 66/135] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:22.097287 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e5eac2-38ba-437f-9fe5-5782dfeb7063 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:22.208515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e5eac2-38ba-437f-9fe5-5782dfeb7063 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:22.243456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26695aaa-04d2-4afb-a61e-ff2af71e614d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:22.244086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26695aaa-04d2-4afb-a61e-ff2af71e614d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:22.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26695aaa-04d2-4afb-a61e-ff2af71e614d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:22.244632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26695aaa-04d2-4afb-a61e-ff2af71e614d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:22.254900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26695aaa-04d2-4afb-a61e-ff2af71e614d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-26695aaa-04d2-4afb-a61e-ff2af71e614d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:22.259711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:22.260518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:22.261326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 70/136] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:22 2022] GET /identity/v3/roles/reader => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:22.268036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc61a25-fa91-4162-8554-013e05dfe4f4 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:22.295732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:22.373155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc61a25-fa91-4162-8554-013e05dfe4f4 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:22.467686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:22.468305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:22.468586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:22.468861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:22.469542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:22.469846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:22.471287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:22.480031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 67/137] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:22 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:22.485813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26695aaa-04d2-4afb-a61e-ff2af71e614d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:22.530269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:22.596728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26695aaa-04d2-4afb-a61e-ff2af71e614d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:22.631766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ebff91-4501-447d-aac2-40ec500c72de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:22.632322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ebff91-4501-447d-aac2-40ec500c72de None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:22.632601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ebff91-4501-447d-aac2-40ec500c72de None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:22.632878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ebff91-4501-447d-aac2-40ec500c72de None admin] PATH_INFO: `/v3/users/06564f22f17a4b3697b3d27f62ee1e7d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:22.634009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ebff91-4501-447d-aac2-40ec500c72de None admin] RBAC: Authorizing `identity:get_user(user_id=06564f22f17a4b3697b3d27f62ee1e7d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:22.635487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ebff91-4501-447d-aac2-40ec500c72de None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:22.637940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 71/138] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:22 2022] GET /identity/v3/users/06564f22f17a4b3697b3d27f62ee1e7d => generated 289 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:22.647194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:22.740063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:22.752404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c827a7-557b-4477-a1dd-0b1ad66b61d4 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:22.796358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c243d41a-a791-4d28-8634-b908572a9d4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:22.796993 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c243d41a-a791-4d28-8634-b908572a9d4e None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:22.797278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c243d41a-a791-4d28-8634-b908572a9d4e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:22.797557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c243d41a-a791-4d28-8634-b908572a9d4e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:22.864633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c243d41a-a791-4d28-8634-b908572a9d4e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:22.866176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c243d41a-a791-4d28-8634-b908572a9d4e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:22.951671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:22.965026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 68/139] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:22.970860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ebff91-4501-447d-aac2-40ec500c72de None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:23.079832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ebff91-4501-447d-aac2-40ec500c72de None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:23.089963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:23.115012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136d4803-0f00-4167-bddb-10f3aa2c469b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:23.115643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d4803-0f00-4167-bddb-10f3aa2c469b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:23.115929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d4803-0f00-4167-bddb-10f3aa2c469b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:23.116213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136d4803-0f00-4167-bddb-10f3aa2c469b None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:23.117353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136d4803-0f00-4167-bddb-10f3aa2c469b None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:23.118830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136d4803-0f00-4167-bddb-10f3aa2c469b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:23.121360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 72/140] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:22 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:23.127162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c243d41a-a791-4d28-8634-b908572a9d4e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:23.234813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c243d41a-a791-4d28-8634-b908572a9d4e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:23.269754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af91203a-8afc-43e1-a271-accdc3a1ee55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:23.271115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91203a-8afc-43e1-a271-accdc3a1ee55 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:23.271605 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91203a-8afc-43e1-a271-accdc3a1ee55 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:23.272117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91203a-8afc-43e1-a271-accdc3a1ee55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:23.273030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91203a-8afc-43e1-a271-accdc3a1ee55 None admin] RBAC: Adding query filter params (role.id=2d48ee5817eb4e4388e1a55c7db0b40d, scope.project.id=ebd577f63e0b41edb25465e82d9af4df, user.id=06564f22f17a4b3697b3d27f62ee1e7d) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:23.273524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91203a-8afc-43e1-a271-accdc3a1ee55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:23.275176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91203a-8afc-43e1-a271-accdc3a1ee55 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:23.281738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 69/141] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:23 2022] GET /identity/v3/role_assignments?user.id=06564f22f17a4b3697b3d27f62ee1e7d&scope.project.id=ebd577f63e0b41edb25465e82d9af4df&role.id=2d48ee5817eb4e4388e1a55c7db0b40d => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:23.902149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938ed9-5d42-422f-b1d6-59189e6b22c3 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:23.902436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938ed9-5d42-422f-b1d6-59189e6b22c3 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:23.902884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53938ed9-5d42-422f-b1d6-59189e6b22c3 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:23.904497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 73/142] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:23.910153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeac1ea-7dd9-4cec-9b4d-36269dfbdcb9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:23.910434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeac1ea-7dd9-4cec-9b4d-36269dfbdcb9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:23.910873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eeac1ea-7dd9-4cec-9b4d-36269dfbdcb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:23.931264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-2eeac1ea-7dd9-4cec-9b4d-36269dfbdcb9 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:23.932477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2eeac1ea-7dd9-4cec-9b4d-36269dfbdcb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:23.983075 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:24.005308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 70/143] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:24.026174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bcd8e9-1218-424d-9fce-b3250e416ee9 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.026516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bcd8e9-1218-424d-9fce-b3250e416ee9 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.026850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2bcd8e9-1218-424d-9fce-b3250e416ee9 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.028346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 74/144] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:24.033287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d975599-72d5-47c8-a35d-0a2c6c2cf6c3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.034008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d975599-72d5-47c8-a35d-0a2c6c2cf6c3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.034642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d975599-72d5-47c8-a35d-0a2c6c2cf6c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.053526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5d975599-72d5-47c8-a35d-0a2c6c2cf6c3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:24.054637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5d975599-72d5-47c8-a35d-0a2c6c2cf6c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:24.095761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c0df41-4b0c-43d7-97fe-53c763d5ba89 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.096066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c0df41-4b0c-43d7-97fe-53c763d5ba89 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.096315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c0df41-4b0c-43d7-97fe-53c763d5ba89 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.097786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 75/145] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:42:24.102411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d294b4-e30c-4811-a19a-7658d89e9da1 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.102701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d294b4-e30c-4811-a19a-7658d89e9da1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.103002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d294b4-e30c-4811-a19a-7658d89e9da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.121303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f9d294b4-e30c-4811-a19a-7658d89e9da1 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:24.122490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f9d294b4-e30c-4811-a19a-7658d89e9da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:24.125551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 71/146] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:24.193397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 76/147] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:24.280459 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d975599-72d5-47c8-a35d-0a2c6c2cf6c3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.375532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d975599-72d5-47c8-a35d-0a2c6c2cf6c3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:24.388138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:24.397391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d294b4-e30c-4811-a19a-7658d89e9da1 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.409647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa855e01-d912-4ec5-94ad-6b348ab5b6e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:24.410354 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa855e01-d912-4ec5-94ad-6b348ab5b6e8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.410609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa855e01-d912-4ec5-94ad-6b348ab5b6e8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.410880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa855e01-d912-4ec5-94ad-6b348ab5b6e8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.422294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa855e01-d912-4ec5-94ad-6b348ab5b6e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-fa855e01-d912-4ec5-94ad-6b348ab5b6e8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:24.428106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:24.429154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:24.429994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:24.431886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 72/148] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:42:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:24.438197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa855e01-d912-4ec5-94ad-6b348ab5b6e8 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.500054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d294b4-e30c-4811-a19a-7658d89e9da1 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:24.507232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:24.533085 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7618eec0-4893-42a4-baed-0cf61a7ea1ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:24.533714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618eec0-4893-42a4-baed-0cf61a7ea1ea None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.533989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618eec0-4893-42a4-baed-0cf61a7ea1ea None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.534261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7618eec0-4893-42a4-baed-0cf61a7ea1ea None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.536069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa855e01-d912-4ec5-94ad-6b348ab5b6e8 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:24.544979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7618eec0-4893-42a4-baed-0cf61a7ea1ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-7618eec0-4893-42a4-baed-0cf61a7ea1ea None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:24.549787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:24.550749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 77/149] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:42:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:24.551677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7618eec0-4893-42a4-baed-0cf61a7ea1ea None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.572189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:24.572797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.573076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.573376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.574094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:24.574398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:24.575913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:24.584059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 73/150] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:24.584059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.619816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:24.653865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7618eec0-4893-42a4-baed-0cf61a7ea1ea None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:24.689147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e3763-371e-4ce1-bdf5-0146bfac0716 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:24.690543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec31d8d1-908c-452d-bf4e-abe087bef968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:24.691198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31d8d1-908c-452d-bf4e-abe087bef968 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.691480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31d8d1-908c-452d-bf4e-abe087bef968 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.691751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31d8d1-908c-452d-bf4e-abe087bef968 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.692863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec31d8d1-908c-452d-bf4e-abe087bef968 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:24.694330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec31d8d1-908c-452d-bf4e-abe087bef968 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:24.695938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 78/151] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:24.696885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec31d8d1-908c-452d-bf4e-abe087bef968 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.722813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:24.723544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.723826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.724097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.724798 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:24.725103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:24.726499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:24.731993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:24.735049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 74/152] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:42:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:24.735850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.806392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec31d8d1-908c-452d-bf4e-abe087bef968 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:24.827285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:24.833126 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c6d150-e944-4b90-a9ad-2f707615c606 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:24.838794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddcf80d-7d2b-4442-9ac7-0146c9f6044a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:24.839993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddcf80d-7d2b-4442-9ac7-0146c9f6044a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.840283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddcf80d-7d2b-4442-9ac7-0146c9f6044a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.840557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddcf80d-7d2b-4442-9ac7-0146c9f6044a None admin] PATH_INFO: `/v3/users/39ae32763d444f3e9aa76ea4b3674f99` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.853353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddcf80d-7d2b-4442-9ac7-0146c9f6044a None admin] RBAC: Authorizing `identity:get_user(user_id=39ae32763d444f3e9aa76ea4b3674f99)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:24.855396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddcf80d-7d2b-4442-9ac7-0146c9f6044a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:24.861513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 79/153] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:24 2022] GET /identity/v3/users/39ae32763d444f3e9aa76ea4b3674f99 => generated 283 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:24.861513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddcf80d-7d2b-4442-9ac7-0146c9f6044a None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.867374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cacac0-bf0f-40d8-976b-b6a32c94e7ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:24.867975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cacac0-bf0f-40d8-976b-b6a32c94e7ea None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.868252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cacac0-bf0f-40d8-976b-b6a32c94e7ea None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.868516 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cacac0-bf0f-40d8-976b-b6a32c94e7ea None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.869154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cacac0-bf0f-40d8-976b-b6a32c94e7ea None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:24.870625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0cacac0-bf0f-40d8-976b-b6a32c94e7ea None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:24.945639 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:24.953855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddcf80d-7d2b-4442-9ac7-0146c9f6044a None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:24.971418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 75/154] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:24 2022] POST /identity/v3/users => generated 297 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:24.983171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cacac0-bf0f-40d8-976b-b6a32c94e7ea None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:24.989292 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2188a4c0-200a-491d-955f-92f64093dbd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:24.989953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2188a4c0-200a-491d-955f-92f64093dbd1 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:24.990234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2188a4c0-200a-491d-955f-92f64093dbd1 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:24.990507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2188a4c0-200a-491d-955f-92f64093dbd1 None admin] PATH_INFO: `/v3/users/39ae32763d444f3e9aa76ea4b3674f99` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:24.991780 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2188a4c0-200a-491d-955f-92f64093dbd1 None admin] RBAC: Authorizing `identity:get_user(user_id=39ae32763d444f3e9aa76ea4b3674f99)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:24.993367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2188a4c0-200a-491d-955f-92f64093dbd1 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:25.001171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 80/155] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:24 2022] GET /identity/v3/users/39ae32763d444f3e9aa76ea4b3674f99 => generated 283 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:25.002610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2188a4c0-200a-491d-955f-92f64093dbd1 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:25.099545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cacac0-bf0f-40d8-976b-b6a32c94e7ea None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:25.121016 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2188a4c0-200a-491d-955f-92f64093dbd1 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:25.135614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba41daf-173b-41f2-bf02-d6ccaa607eef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:25.136938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba41daf-173b-41f2-bf02-d6ccaa607eef None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:25.137195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba41daf-173b-41f2-bf02-d6ccaa607eef None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:25.137466 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fba41daf-173b-41f2-bf02-d6ccaa607eef None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:25.139116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba41daf-173b-41f2-bf02-d6ccaa607eef None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:25.140135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fba41daf-173b-41f2-bf02-d6ccaa607eef None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:25.141865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 76/156] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:24 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:25.148797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba41daf-173b-41f2-bf02-d6ccaa607eef None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:25.171736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:25.172114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eaac2c-1427-4010-aabe-e240d179fe44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:25.172886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eaac2c-1427-4010-aabe-e240d179fe44 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:25.173320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eaac2c-1427-4010-aabe-e240d179fe44 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:25.173971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15eaac2c-1427-4010-aabe-e240d179fe44 None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:25.175366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15eaac2c-1427-4010-aabe-e240d179fe44 None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:25.176867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15eaac2c-1427-4010-aabe-e240d179fe44 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:25.178514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 81/157] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:25 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:25.184813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eaac2c-1427-4010-aabe-e240d179fe44 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:25.215282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:25.263846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fba41daf-173b-41f2-bf02-d6ccaa607eef None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:25.303635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15eaac2c-1427-4010-aabe-e240d179fe44 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:25.308889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd87deb2-af72-4126-ab31-46f91ca9839c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:25.310139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd87deb2-af72-4126-ab31-46f91ca9839c None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:25.311022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd87deb2-af72-4126-ab31-46f91ca9839c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:25.311785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd87deb2-af72-4126-ab31-46f91ca9839c None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/users/39ae32763d444f3e9aa76ea4b3674f99/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:25.313773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd87deb2-af72-4126-ab31-46f91ca9839c None admin] RBAC: Authorizing `identity:create_grant(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18, user_id=39ae32763d444f3e9aa76ea4b3674f99, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:25.315256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd87deb2-af72-4126-ab31-46f91ca9839c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:25.339898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea31e32-3c20-4b04-90d7-c9d620b7e5d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:25.340563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea31e32-3c20-4b04-90d7-c9d620b7e5d6 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:25.340844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea31e32-3c20-4b04-90d7-c9d620b7e5d6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:25.341114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea31e32-3c20-4b04-90d7-c9d620b7e5d6 None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/users/39ae32763d444f3e9aa76ea4b3674f99/roles/3cb31ad012324a57acd3a4bd258d1fb1` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:25.343537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea31e32-3c20-4b04-90d7-c9d620b7e5d6 None admin] RBAC: Authorizing `identity:create_grant(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18, user_id=39ae32763d444f3e9aa76ea4b3674f99, role_id=3cb31ad012324a57acd3a4bd258d1fb1)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:25.345326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea31e32-3c20-4b04-90d7-c9d620b7e5d6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:25.378222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 77/158] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:25 2022] PUT /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/users/39ae32763d444f3e9aa76ea4b3674f99/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 21:42:25.430139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 82/159] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:25 2022] PUT /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/users/39ae32763d444f3e9aa76ea4b3674f99/roles/3cb31ad012324a57acd3a4bd258d1fb1 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 21:42:27.293908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1a9794-ea71-4f29-bb14-9adc09591999 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.294450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1a9794-ea71-4f29-bb14-9adc09591999 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.294450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1a9794-ea71-4f29-bb14-9adc09591999 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.295847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 78/160] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:27.301176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a489fd89-0258-4dc6-9d5d-c9d0238d38d0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.301583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a489fd89-0258-4dc6-9d5d-c9d0238d38d0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.301774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a489fd89-0258-4dc6-9d5d-c9d0238d38d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.318381 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f9297-cde5-417f-b374-25c69777f95e None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.319003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f9297-cde5-417f-b374-25c69777f95e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.319486 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2f9297-cde5-417f-b374-25c69777f95e None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.320925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a489fd89-0258-4dc6-9d5d-c9d0238d38d0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:27.321235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 79/161] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:42:27.322127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a489fd89-0258-4dc6-9d5d-c9d0238d38d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:27.326602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff8827c-a2c3-4a4e-a501-bd9bf247a1c0 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.326882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff8827c-a2c3-4a4e-a501-bd9bf247a1c0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.327188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff8827c-a2c3-4a4e-a501-bd9bf247a1c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.352678 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8ff8827c-a2c3-4a4e-a501-bd9bf247a1c0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:27.355579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8ff8827c-a2c3-4a4e-a501-bd9bf247a1c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:27.395907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 83/162] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:27.437892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 80/163] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:27.552660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d6d9f-e6a1-4bb6-883f-a9810c050b18 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.552862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d6d9f-e6a1-4bb6-883f-a9810c050b18 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.553143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2d6d9f-e6a1-4bb6-883f-a9810c050b18 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.554488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 84/164] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:27.560395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2b1ac-8379-4017-a66a-d8f4af3bf3f9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.560782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2b1ac-8379-4017-a66a-d8f4af3bf3f9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.561176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c2b1ac-8379-4017-a66a-d8f4af3bf3f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.580036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-46c2b1ac-8379-4017-a66a-d8f4af3bf3f9 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:27.581205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-46c2b1ac-8379-4017-a66a-d8f4af3bf3f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:27.584842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:27.623649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babc9bf8-997c-49e8-8431-bf7a54906eed None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.623967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babc9bf8-997c-49e8-8431-bf7a54906eed None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.624209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-babc9bf8-997c-49e8-8431-bf7a54906eed None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.625794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 85/165] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:27.630923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20aba23-38a0-48b0-be18-67b70b9a8501 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.631233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20aba23-38a0-48b0-be18-67b70b9a8501 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.631513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20aba23-38a0-48b0-be18-67b70b9a8501 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.650088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e20aba23-38a0-48b0-be18-67b70b9a8501 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:27.651235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 81/166] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:27.652494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e20aba23-38a0-48b0-be18-67b70b9a8501 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:27.666730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:27.706271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c2b1ac-8379-4017-a66a-d8f4af3bf3f9 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:27.732840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 86/167] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:27.733690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20aba23-38a0-48b0-be18-67b70b9a8501 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:27.819329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c2b1ac-8379-4017-a66a-d8f4af3bf3f9 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:27.856333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20aba23-38a0-48b0-be18-67b70b9a8501 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:27.862685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb065fc6-e13b-48ba-982f-adfe63546f51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:27.863453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb065fc6-e13b-48ba-982f-adfe63546f51 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.863826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb065fc6-e13b-48ba-982f-adfe63546f51 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.864190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb065fc6-e13b-48ba-982f-adfe63546f51 None admin] PATH_INFO: `/v3/roles/member` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.875564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:27.879300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb065fc6-e13b-48ba-982f-adfe63546f51 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-eb065fc6-e13b-48ba-982f-adfe63546f51 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:27.887055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:27.887977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:27.892456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 82/168] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:27.895889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb065fc6-e13b-48ba-982f-adfe63546f51 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:27.899876 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cb5a05-9610-44ff-8d28-a6de6f686033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:27.902092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cb5a05-9610-44ff-8d28-a6de6f686033 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:27.902384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cb5a05-9610-44ff-8d28-a6de6f686033 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:27.902662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cb5a05-9610-44ff-8d28-a6de6f686033 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:27.903810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cb5a05-9610-44ff-8d28-a6de6f686033 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:27.905261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cb5a05-9610-44ff-8d28-a6de6f686033 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:27.907163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 87/169] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:27.912646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cb5a05-9610-44ff-8d28-a6de6f686033 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:27.937886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:28.004201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb065fc6-e13b-48ba-982f-adfe63546f51 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.027169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cb5a05-9610-44ff-8d28-a6de6f686033 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.037368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.037972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.038244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.038566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.039263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] RBAC: Adding query filter params (name=member) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:28.039563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.040955 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.049184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 83/170] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:28.049957 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.062027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9310bfaa-1239-406c-afe9-66eb0f5caa5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.062689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310bfaa-1239-406c-afe9-66eb0f5caa5b None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.062991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310bfaa-1239-406c-afe9-66eb0f5caa5b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.063288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9310bfaa-1239-406c-afe9-66eb0f5caa5b None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.063922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9310bfaa-1239-406c-afe9-66eb0f5caa5b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.065363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9310bfaa-1239-406c-afe9-66eb0f5caa5b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.112353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:28.148882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-444f05ff-a722-4c0c-baa4-35ae3aa66577 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.153823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 88/171] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:27 2022] POST /identity/v3/users => generated 297 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:42:28.154659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9310bfaa-1239-406c-afe9-66eb0f5caa5b None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.193442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5196f16-6c98-4cce-a537-3d840887b08c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.194157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5196f16-6c98-4cce-a537-3d840887b08c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.194366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5196f16-6c98-4cce-a537-3d840887b08c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.194638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5196f16-6c98-4cce-a537-3d840887b08c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.206667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5196f16-6c98-4cce-a537-3d840887b08c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c5196f16-6c98-4cce-a537-3d840887b08c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:28.212437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:28.213395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:28.214154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 84/172] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:28.214800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5196f16-6c98-4cce-a537-3d840887b08c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.255952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:28.286184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9310bfaa-1239-406c-afe9-66eb0f5caa5b None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.325926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa93e18-d12c-4737-b503-b3590d87f843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.327415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa93e18-d12c-4737-b503-b3590d87f843 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.327865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa93e18-d12c-4737-b503-b3590d87f843 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.328372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa93e18-d12c-4737-b503-b3590d87f843 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.336016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5196f16-6c98-4cce-a537-3d840887b08c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.339507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa93e18-d12c-4737-b503-b3590d87f843 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-caa93e18-d12c-4737-b503-b3590d87f843 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:28.344502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:28.345383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:28.346172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 89/173] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:28.346802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa93e18-d12c-4737-b503-b3590d87f843 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.370221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:28.378915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27b0e45-42d1-4649-8ec9-f888dadd7997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.379674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b0e45-42d1-4649-8ec9-f888dadd7997 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.379958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b0e45-42d1-4649-8ec9-f888dadd7997 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.380229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27b0e45-42d1-4649-8ec9-f888dadd7997 None admin] PATH_INFO: `/v3/users/9377a7f18c8f42afb62301059022f6da` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.394155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27b0e45-42d1-4649-8ec9-f888dadd7997 None admin] RBAC: Authorizing `identity:get_user(user_id=9377a7f18c8f42afb62301059022f6da)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.395818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27b0e45-42d1-4649-8ec9-f888dadd7997 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.398864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 85/174] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/users/9377a7f18c8f42afb62301059022f6da => generated 297 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:28.398864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27b0e45-42d1-4649-8ec9-f888dadd7997 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.470998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa93e18-d12c-4737-b503-b3590d87f843 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.509202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:28.515070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.516169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.516477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27b0e45-42d1-4649-8ec9-f888dadd7997 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.516584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.516762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.517438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] RBAC: Adding query filter params (name=admin) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:28.517738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.519195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.529179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 90/175] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:28.531812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.559000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.559880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.560301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.560715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.561530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:28.561975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.563552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.576205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 86/176] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:28.577281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.622189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:28.647836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb01be30-0e27-4d04-9404-d9250267abec None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.686349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d34ff5-f30f-4cef-9f05-afffa607c134 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.688622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023036df-133c-40f8-90a9-cd676a04ed9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.689453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023036df-133c-40f8-90a9-cd676a04ed9a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.689800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023036df-133c-40f8-90a9-cd676a04ed9a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.690153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023036df-133c-40f8-90a9-cd676a04ed9a None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.691408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023036df-133c-40f8-90a9-cd676a04ed9a None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.692970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023036df-133c-40f8-90a9-cd676a04ed9a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.694878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 91/177] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:28.704941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023036df-133c-40f8-90a9-cd676a04ed9a None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.730034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a443da-28c1-4ba5-a995-e5c04f03c0a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.730701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a443da-28c1-4ba5-a995-e5c04f03c0a8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.731013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a443da-28c1-4ba5-a995-e5c04f03c0a8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.731292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a443da-28c1-4ba5-a995-e5c04f03c0a8 None admin] PATH_INFO: `/v3/users/39ae32763d444f3e9aa76ea4b3674f99` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.732554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a443da-28c1-4ba5-a995-e5c04f03c0a8 None admin] RBAC: Authorizing `identity:get_user(user_id=39ae32763d444f3e9aa76ea4b3674f99)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.734182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6a443da-28c1-4ba5-a995-e5c04f03c0a8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.735992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 87/178] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/users/39ae32763d444f3e9aa76ea4b3674f99 => generated 283 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:28.736850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a443da-28c1-4ba5-a995-e5c04f03c0a8 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.790287 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:28.837109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023036df-133c-40f8-90a9-cd676a04ed9a None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.869719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6a443da-28c1-4ba5-a995-e5c04f03c0a8 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:28.883343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:28.885127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f956e6e2-3777-406a-86fa-cef47e7c12b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.886283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f956e6e2-3777-406a-86fa-cef47e7c12b1 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.886573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f956e6e2-3777-406a-86fa-cef47e7c12b1 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.886843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f956e6e2-3777-406a-86fa-cef47e7c12b1 None admin] PATH_INFO: `/v3/users/39ae32763d444f3e9aa76ea4b3674f99` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.888103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f956e6e2-3777-406a-86fa-cef47e7c12b1 None admin] RBAC: Authorizing `identity:get_user(user_id=39ae32763d444f3e9aa76ea4b3674f99)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.889546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f956e6e2-3777-406a-86fa-cef47e7c12b1 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.895701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 92/179] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/users/39ae32763d444f3e9aa76ea4b3674f99 => generated 283 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:28.895701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f956e6e2-3777-406a-86fa-cef47e7c12b1 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:28.915519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a227f6-9793-4593-afbc-0ed1751ac7fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:28.916523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a227f6-9793-4593-afbc-0ed1751ac7fa None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:28.916948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a227f6-9793-4593-afbc-0ed1751ac7fa None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:28.917392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a227f6-9793-4593-afbc-0ed1751ac7fa None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/users/9377a7f18c8f42afb62301059022f6da/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:28.919701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a227f6-9793-4593-afbc-0ed1751ac7fa None admin] RBAC: Authorizing `identity:create_grant(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18, user_id=9377a7f18c8f42afb62301059022f6da, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:28.921304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a227f6-9793-4593-afbc-0ed1751ac7fa None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:28.977945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 88/180] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:28 2022] PUT /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/users/9377a7f18c8f42afb62301059022f6da/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:28.980257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a227f6-9793-4593-afbc-0ed1751ac7fa None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:29.033768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f956e6e2-3777-406a-86fa-cef47e7c12b1 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:29.102925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:29.115986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83e8627-aaa7-4053-8ae8-fd461f5f03fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:29.118084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83e8627-aaa7-4053-8ae8-fd461f5f03fa None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:29.119173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83e8627-aaa7-4053-8ae8-fd461f5f03fa None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:29.120094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83e8627-aaa7-4053-8ae8-fd461f5f03fa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:29.181458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a227f6-9793-4593-afbc-0ed1751ac7fa None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:29.226101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83e8627-aaa7-4053-8ae8-fd461f5f03fa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:29.227845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83e8627-aaa7-4053-8ae8-fd461f5f03fa None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:29.230447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0384dc-442a-4cba-b6cd-6256bf5a0fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:29.232480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0384dc-442a-4cba-b6cd-6256bf5a0fc3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:29.233226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0384dc-442a-4cba-b6cd-6256bf5a0fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:29.233912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0384dc-442a-4cba-b6cd-6256bf5a0fc3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:29.291339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:29.329937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0384dc-442a-4cba-b6cd-6256bf5a0fc3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:29.334946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d0384dc-442a-4cba-b6cd-6256bf5a0fc3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:29.410612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 93/181] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 518 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:29.416202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83e8627-aaa7-4053-8ae8-fd461f5f03fa None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:29.451138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:29.526859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83e8627-aaa7-4053-8ae8-fd461f5f03fa None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:29.548780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 89/182] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 566 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:29.551943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0384dc-442a-4cba-b6cd-6256bf5a0fc3 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:29.589812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d9dbe8-7893-4471-bcaa-0d17f8e2500e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:29.590812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d9dbe8-7893-4471-bcaa-0d17f8e2500e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:29.591351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d9dbe8-7893-4471-bcaa-0d17f8e2500e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:29.592041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d9dbe8-7893-4471-bcaa-0d17f8e2500e None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:29.594460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d9dbe8-7893-4471-bcaa-0d17f8e2500e None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:29.597203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24d9dbe8-7893-4471-bcaa-0d17f8e2500e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:29.600458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 94/183] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:29 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:29.606736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d9dbe8-7893-4471-bcaa-0d17f8e2500e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:29.693752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:29.718823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d9dbe8-7893-4471-bcaa-0d17f8e2500e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:29.739209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:29.756717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0384dc-442a-4cba-b6cd-6256bf5a0fc3 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:29.759565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:29.760442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:29.761471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:29.761953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:29.762900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18, user.id=39ae32763d444f3e9aa76ea4b3674f99) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:29.763459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:29.765116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:29.771481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 95/184] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:29 2022] GET /identity/v3/role_assignments?user.id=39ae32763d444f3e9aa76ea4b3674f99&scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:29.835669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a13e4f-3221-447f-856b-a7781bcadca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:29.836314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a13e4f-3221-447f-856b-a7781bcadca3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:29.836589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a13e4f-3221-447f-856b-a7781bcadca3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:29.836858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a13e4f-3221-447f-856b-a7781bcadca3 None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:29.838071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a13e4f-3221-447f-856b-a7781bcadca3 None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:29.840179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54a13e4f-3221-447f-856b-a7781bcadca3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:29.841982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 90/185] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:29 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:29.848008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:29.983680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b8f0f-413f-4ea2-b2b5-e045c64a10e9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:30.023415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72f538a-93be-4f25-9c01-1c32ef6ef2b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:30.024024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f538a-93be-4f25-9c01-1c32ef6ef2b6 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:30.024301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f538a-93be-4f25-9c01-1c32ef6ef2b6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:30.024570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72f538a-93be-4f25-9c01-1c32ef6ef2b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:30.025299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72f538a-93be-4f25-9c01-1c32ef6ef2b6 None admin] RBAC: Adding query filter params (role.id=3cb31ad012324a57acd3a4bd258d1fb1, scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18, user.id=39ae32763d444f3e9aa76ea4b3674f99) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:30.025604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72f538a-93be-4f25-9c01-1c32ef6ef2b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:30.027375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72f538a-93be-4f25-9c01-1c32ef6ef2b6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:30.033570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 96/186] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:29 2022] GET /identity/v3/role_assignments?user.id=39ae32763d444f3e9aa76ea4b3674f99&scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18&role.id=3cb31ad012324a57acd3a4bd258d1fb1 => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:30.668553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f54cb-538a-417b-8f77-5eb854c94550 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:30.669099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f54cb-538a-417b-8f77-5eb854c94550 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:30.669099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096f54cb-538a-417b-8f77-5eb854c94550 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:30.670470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 91/187] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:30.675670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a046dd-df7b-4f26-9d62-dbfed5b73978 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:30.676090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a046dd-df7b-4f26-9d62-dbfed5b73978 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:30.676254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a046dd-df7b-4f26-9d62-dbfed5b73978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:30.694169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-67a046dd-df7b-4f26-9d62-dbfed5b73978 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:30.695401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-67a046dd-df7b-4f26-9d62-dbfed5b73978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:30.710097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b83b0d3-5084-4a42-b7a1-1f623514e128 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:30.710377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b83b0d3-5084-4a42-b7a1-1f623514e128 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:30.710652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b83b0d3-5084-4a42-b7a1-1f623514e128 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:30.712189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 92/188] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:30.717268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:30.717396 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de401b6-6a26-4fc7-a824-d73a000de7db None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:30.717561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de401b6-6a26-4fc7-a824-d73a000de7db None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:30.717828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de401b6-6a26-4fc7-a824-d73a000de7db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:30.740234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8de401b6-6a26-4fc7-a824-d73a000de7db None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:30.741475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8de401b6-6a26-4fc7-a824-d73a000de7db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:30.771659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 97/189] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:30.813866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 93/190] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:31.036807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a046dd-df7b-4f26-9d62-dbfed5b73978 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:31.079535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de401b6-6a26-4fc7-a824-d73a000de7db None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:31.140304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a046dd-df7b-4f26-9d62-dbfed5b73978 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:31.148429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:31.175424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ff9f06-769d-43c3-867b-311a90fb2072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:31.176059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ff9f06-769d-43c3-867b-311a90fb2072 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.176332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ff9f06-769d-43c3-867b-311a90fb2072 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.176600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ff9f06-769d-43c3-867b-311a90fb2072 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.186502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de401b6-6a26-4fc7-a824-d73a000de7db None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:31.186860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6ff9f06-769d-43c3-867b-311a90fb2072 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-b6ff9f06-769d-43c3-867b-311a90fb2072 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:31.191769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:31.192888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:31.193899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 98/191] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:31.200958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ff9f06-769d-43c3-867b-311a90fb2072 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:31.225479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:31.247770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b759f18-35aa-40fb-a1e0-efb588261442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:31.248579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b759f18-35aa-40fb-a1e0-efb588261442 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.248910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b759f18-35aa-40fb-a1e0-efb588261442 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.249264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b759f18-35aa-40fb-a1e0-efb588261442 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.250512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b759f18-35aa-40fb-a1e0-efb588261442 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:31.251973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b759f18-35aa-40fb-a1e0-efb588261442 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:31.253724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 94/192] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:31.259318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b759f18-35aa-40fb-a1e0-efb588261442 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:31.322618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ff9f06-769d-43c3-867b-311a90fb2072 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:31.367541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:31.368160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.368431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.368705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.369376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:31.369679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:31.371761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:31.380878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 99/193] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:31.386390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:31.427151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b759f18-35aa-40fb-a1e0-efb588261442 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:31.474135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ab30d4-51b0-450e-8ec6-0bc24257aa30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:31.474741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab30d4-51b0-450e-8ec6-0bc24257aa30 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.475043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab30d4-51b0-450e-8ec6-0bc24257aa30 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.475317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ab30d4-51b0-450e-8ec6-0bc24257aa30 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.475978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ab30d4-51b0-450e-8ec6-0bc24257aa30 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:31.477422 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ab30d4-51b0-450e-8ec6-0bc24257aa30 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:31.487164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:31.487812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:31.511882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da86c0-6c5b-41e3-9db6-bec28c22cff6 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:31.546142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e56d34-5b5d-4597-8720-e98b0cda6e68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:31.546776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e56d34-5b5d-4597-8720-e98b0cda6e68 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.547083 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e56d34-5b5d-4597-8720-e98b0cda6e68 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.550729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e56d34-5b5d-4597-8720-e98b0cda6e68 None admin] PATH_INFO: `/v3/users/0d61dff869894d7897f5ad3461a19e7c` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.560224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e56d34-5b5d-4597-8720-e98b0cda6e68 None admin] RBAC: Authorizing `identity:get_user(user_id=0d61dff869894d7897f5ad3461a19e7c)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:31.561676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e56d34-5b5d-4597-8720-e98b0cda6e68 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:31.563307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 100/194] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:31 2022] GET /identity/v3/users/0d61dff869894d7897f5ad3461a19e7c => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:31.564451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf126ee-162d-457e-97a1-acecedd1eae2 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.564733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf126ee-162d-457e-97a1-acecedd1eae2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.565005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf126ee-162d-457e-97a1-acecedd1eae2 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.566202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 101/195] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:31 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:31.569120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf126ee-162d-457e-97a1-acecedd1eae2 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:31.582378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 95/196] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:31 2022] POST /identity/v3/users => generated 293 bytes in 323 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:31.583571 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31204b17-fc41-433b-8e4b-610f10f3f76b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.583860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31204b17-fc41-433b-8e4b-610f10f3f76b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.584146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31204b17-fc41-433b-8e4b-610f10f3f76b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.603662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-31204b17-fc41-433b-8e4b-610f10f3f76b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:31.605484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-31204b17-fc41-433b-8e4b-610f10f3f76b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:31.678883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 96/197] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:31.685912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf126ee-162d-457e-97a1-acecedd1eae2 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:31.722507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb1d529-9107-43e2-88f3-d0c2c9b6fc0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:31.723183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1d529-9107-43e2-88f3-d0c2c9b6fc0e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.723464 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1d529-9107-43e2-88f3-d0c2c9b6fc0e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.723738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb1d529-9107-43e2-88f3-d0c2c9b6fc0e None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.724922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb1d529-9107-43e2-88f3-d0c2c9b6fc0e None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:31.726375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb1d529-9107-43e2-88f3-d0c2c9b6fc0e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:31.728276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 102/198] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:31 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:31.733975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31204b17-fc41-433b-8e4b-610f10f3f76b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:31.843794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31204b17-fc41-433b-8e4b-610f10f3f76b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:31.868066 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:31.881646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dfac20-cf53-4215-97d1-9b79142101a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:31.885315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dfac20-cf53-4215-97d1-9b79142101a9 None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:31.885315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dfac20-cf53-4215-97d1-9b79142101a9 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:31.885315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dfac20-cf53-4215-97d1-9b79142101a9 None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/users/0d61dff869894d7897f5ad3461a19e7c/roles/2d48ee5817eb4e4388e1a55c7db0b40d` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:31.885315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dfac20-cf53-4215-97d1-9b79142101a9 None admin] RBAC: Authorizing `identity:create_grant(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18, user_id=0d61dff869894d7897f5ad3461a19e7c, role_id=2d48ee5817eb4e4388e1a55c7db0b40d)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:31.886328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3dfac20-cf53-4215-97d1-9b79142101a9 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:31.953031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb1d529-9107-43e2-88f3-d0c2c9b6fc0e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:31.957336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:31.961048 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 97/199] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:42:31 2022] PUT /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/users/0d61dff869894d7897f5ad3461a19e7c/roles/2d48ee5817eb4e4388e1a55c7db0b40d => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:32.070896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb1d529-9107-43e2-88f3-d0c2c9b6fc0e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:32.109881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455e88b-0959-4bab-b0e9-2069b446a7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:32.110455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455e88b-0959-4bab-b0e9-2069b446a7f9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:32.110730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455e88b-0959-4bab-b0e9-2069b446a7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:32.111039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6455e88b-0959-4bab-b0e9-2069b446a7f9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:32.123512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6455e88b-0959-4bab-b0e9-2069b446a7f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-6455e88b-0959-4bab-b0e9-2069b446a7f9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:32.128989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:32.129857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:32.130643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 103/200] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:32.137499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dfac20-cf53-4215-97d1-9b79142101a9 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:32.244648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3dfac20-cf53-4215-97d1-9b79142101a9 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:32.286231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:32.286844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:32.287184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:32.287464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:32.288143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] RBAC: Adding query filter params (name=member) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:32.288510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:32.289906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:32.302994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 98/201] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:32.308727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455e88b-0959-4bab-b0e9-2069b446a7f9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:32.375695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:32.415155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6455e88b-0959-4bab-b0e9-2069b446a7f9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:32.454587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba506c18-538d-481d-8346-f564287f9363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:32.455259 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba506c18-538d-481d-8346-f564287f9363 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:32.455539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba506c18-538d-481d-8346-f564287f9363 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:32.455916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba506c18-538d-481d-8346-f564287f9363 None admin] PATH_INFO: `/v3/users/9377a7f18c8f42afb62301059022f6da` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:32.457051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba506c18-538d-481d-8346-f564287f9363 None admin] RBAC: Authorizing `identity:get_user(user_id=9377a7f18c8f42afb62301059022f6da)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:32.458509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba506c18-538d-481d-8346-f564287f9363 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:32.460117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 104/202] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:32 2022] GET /identity/v3/users/9377a7f18c8f42afb62301059022f6da => generated 297 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:32.466375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:32.518973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:32.576180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda73ae0-be39-4288-b3a3-5faa59c9de11 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:32.612150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4c3a38-e0f3-4259-affc-9908b1338a2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:32.612759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4c3a38-e0f3-4259-affc-9908b1338a2c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:32.613038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4c3a38-e0f3-4259-affc-9908b1338a2c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:32.613312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4c3a38-e0f3-4259-affc-9908b1338a2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:32.694921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4c3a38-e0f3-4259-affc-9908b1338a2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:32.696445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4c3a38-e0f3-4259-affc-9908b1338a2c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:32.744542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:32.810411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 99/203] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:32.815832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba506c18-538d-481d-8346-f564287f9363 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:32.920544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba506c18-538d-481d-8346-f564287f9363 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:32.953608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:32.957771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70876520-565a-46d5-bb29-6c3bcb3d0997 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:32.958648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70876520-565a-46d5-bb29-6c3bcb3d0997 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:32.959187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70876520-565a-46d5-bb29-6c3bcb3d0997 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:32.959671 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70876520-565a-46d5-bb29-6c3bcb3d0997 None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:32.961023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70876520-565a-46d5-bb29-6c3bcb3d0997 None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:32.962752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70876520-565a-46d5-bb29-6c3bcb3d0997 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:32.964710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 105/204] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:32 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:32.970999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4c3a38-e0f3-4259-affc-9908b1338a2c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:33.074249 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4c3a38-e0f3-4259-affc-9908b1338a2c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:33.109112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3dd12d-a3d2-4b74-b0a8-c4efed19f907 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:33.109719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3dd12d-a3d2-4b74-b0a8-c4efed19f907 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:33.109999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3dd12d-a3d2-4b74-b0a8-c4efed19f907 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:33.110275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3dd12d-a3d2-4b74-b0a8-c4efed19f907 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:33.111038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3dd12d-a3d2-4b74-b0a8-c4efed19f907 None admin] RBAC: Adding query filter params (role.id=974299ae4a414939a4f7e186921d4569, scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18, user.id=9377a7f18c8f42afb62301059022f6da) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:33.111375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3dd12d-a3d2-4b74-b0a8-c4efed19f907 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:33.112887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3dd12d-a3d2-4b74-b0a8-c4efed19f907 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:33.118842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 100/205] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:32 2022] GET /identity/v3/role_assignments?user.id=9377a7f18c8f42afb62301059022f6da&scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18&role.id=974299ae4a414939a4f7e186921d4569 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:33.960090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8cd729-01bb-4152-9329-e3b1b7694413 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:33.960604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8cd729-01bb-4152-9329-e3b1b7694413 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:33.960714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8cd729-01bb-4152-9329-e3b1b7694413 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:33.962174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 106/206] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:33.967632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb01a8f-4a49-4e99-b79e-e2e699a783fb None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:33.968087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb01a8f-4a49-4e99-b79e-e2e699a783fb None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:33.968258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb01a8f-4a49-4e99-b79e-e2e699a783fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:33.986319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-2cb01a8f-4a49-4e99-b79e-e2e699a783fb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:33.987527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2cb01a8f-4a49-4e99-b79e-e2e699a783fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:33.989464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:34.048229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9c02d-1d0c-4470-b3f3-90fb898bf72c None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.048428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9c02d-1d0c-4470-b3f3-90fb898bf72c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.048705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff9c02d-1d0c-4470-b3f3-90fb898bf72c None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.050045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 107/207] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:34.055696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195f4438-a966-4b9e-8c4b-901c2ffeedfb None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.055980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195f4438-a966-4b9e-8c4b-901c2ffeedfb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.056256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195f4438-a966-4b9e-8c4b-901c2ffeedfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.059227 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 101/208] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:34.081002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-195f4438-a966-4b9e-8c4b-901c2ffeedfb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:34.082228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-195f4438-a966-4b9e-8c4b-901c2ffeedfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:34.154985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 108/209] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:34.185265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ee5f7f-b7fc-406b-af28-bea76450b025 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.185549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ee5f7f-b7fc-406b-af28-bea76450b025 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.185976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ee5f7f-b7fc-406b-af28-bea76450b025 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.187499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 102/210] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:34.192735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81a1a2a-141a-4c2a-8d38-dd605721fd1b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.193166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81a1a2a-141a-4c2a-8d38-dd605721fd1b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.193597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81a1a2a-141a-4c2a-8d38-dd605721fd1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.211367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d81a1a2a-141a-4c2a-8d38-dd605721fd1b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:34.212694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d81a1a2a-141a-4c2a-8d38-dd605721fd1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:34.280520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 109/211] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:34.331054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ee5f7f-b7fc-406b-af28-bea76450b025 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:34.423114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81a1a2a-141a-4c2a-8d38-dd605721fd1b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:34.467608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ee5f7f-b7fc-406b-af28-bea76450b025 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:34.481462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:34.503129 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28512ee7-139e-4abb-a300-c13f90e04647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:34.503752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28512ee7-139e-4abb-a300-c13f90e04647 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.504024 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28512ee7-139e-4abb-a300-c13f90e04647 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.504295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28512ee7-139e-4abb-a300-c13f90e04647 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.505379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28512ee7-139e-4abb-a300-c13f90e04647 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:34.506824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28512ee7-139e-4abb-a300-c13f90e04647 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:34.511000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 103/212] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:34.514154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28512ee7-139e-4abb-a300-c13f90e04647 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:34.522767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81a1a2a-141a-4c2a-8d38-dd605721fd1b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:34.523281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:34.558429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d872cb02-cc48-4cbb-bb3f-42b91ecbc5b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:34.559083 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d872cb02-cc48-4cbb-bb3f-42b91ecbc5b8 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.559357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d872cb02-cc48-4cbb-bb3f-42b91ecbc5b8 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.559624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d872cb02-cc48-4cbb-bb3f-42b91ecbc5b8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.569562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d872cb02-cc48-4cbb-bb3f-42b91ecbc5b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-d872cb02-cc48-4cbb-bb3f-42b91ecbc5b8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:34.574321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:34.575335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:34.576117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 110/213] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:34.576753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d872cb02-cc48-4cbb-bb3f-42b91ecbc5b8 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:34.616983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28512ee7-139e-4abb-a300-c13f90e04647 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:34.648795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ed5ea4-0486-4f3d-935a-a73dd6ab164e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:34.649400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed5ea4-0486-4f3d-935a-a73dd6ab164e None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.649680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed5ea4-0486-4f3d-935a-a73dd6ab164e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.649953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ed5ea4-0486-4f3d-935a-a73dd6ab164e None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.650612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed5ea4-0486-4f3d-935a-a73dd6ab164e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:34.652870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ed5ea4-0486-4f3d-935a-a73dd6ab164e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:34.673584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d872cb02-cc48-4cbb-bb3f-42b91ecbc5b8 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:34.693258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:34.706873 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8fed51-61d0-4f57-a4de-140139b91212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:34.707518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8fed51-61d0-4f57-a4de-140139b91212 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.707797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8fed51-61d0-4f57-a4de-140139b91212 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.708068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8fed51-61d0-4f57-a4de-140139b91212 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.717636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8fed51-61d0-4f57-a4de-140139b91212 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-8a8fed51-61d0-4f57-a4de-140139b91212 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:34.722418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:34.723509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 111/214] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:34 2022] GET /identity/v3/roles/reader => generated 84 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:34.724300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8fed51-61d0-4f57-a4de-140139b91212 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:34.748865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 104/215] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:34 2022] POST /identity/v3/users => generated 293 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:34.751935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ed5ea4-0486-4f3d-935a-a73dd6ab164e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:34.830580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:34.840997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8fed51-61d0-4f57-a4de-140139b91212 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:34.873261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ed5ea4-0486-4f3d-935a-a73dd6ab164e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:34.884516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:34.885460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.886548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.886861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.887587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] RBAC: Adding query filter params (name=member) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:34.888079 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:34.889513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:34.899795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 112/216] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:34.906528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:34.914798 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:34.915458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:34.915744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:34.916020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:34.916713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] RBAC: Adding query filter params (name=reader) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:34.917530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:34.918971 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:34.928779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 105/217] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:34 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:34.934854 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:34.956372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:35.031241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e37655-f498-4d1e-ac6c-1105b20c4a94 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:35.054579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4077e389-dd45-478d-92c3-2d13893c0c0d None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:35.069955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f0e90-99bd-4054-ab31-3dfcf806c1bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:35.070598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f0e90-99bd-4054-ab31-3dfcf806c1bb None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:35.070880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f0e90-99bd-4054-ab31-3dfcf806c1bb None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:35.071186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23f0e90-99bd-4054-ab31-3dfcf806c1bb None admin] PATH_INFO: `/v3/users/0ae5216cd2814e7e89183ec170c3b8f8` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:35.081059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:35.089852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23f0e90-99bd-4054-ab31-3dfcf806c1bb None admin] RBAC: Authorizing `identity:get_user(user_id=0ae5216cd2814e7e89183ec170c3b8f8)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:35.094005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23f0e90-99bd-4054-ab31-3dfcf806c1bb None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:35.094190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 113/218] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:34 2022] GET /identity/v3/users/0ae5216cd2814e7e89183ec170c3b8f8 => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:35.094285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ec134d-efbf-495b-9e7d-0a3b35f71217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:35.094871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec134d-efbf-495b-9e7d-0a3b35f71217 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:35.095215 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec134d-efbf-495b-9e7d-0a3b35f71217 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:35.095487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ec134d-efbf-495b-9e7d-0a3b35f71217 None admin] PATH_INFO: `/v3/users/0d61dff869894d7897f5ad3461a19e7c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:35.096686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ec134d-efbf-495b-9e7d-0a3b35f71217 None admin] RBAC: Authorizing `identity:get_user(user_id=0d61dff869894d7897f5ad3461a19e7c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:35.098166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ec134d-efbf-495b-9e7d-0a3b35f71217 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:35.099789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 106/219] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:34 2022] GET /identity/v3/users/0d61dff869894d7897f5ad3461a19e7c => generated 297 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:35.100224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f0e90-99bd-4054-ab31-3dfcf806c1bb None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:35.107248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ec134d-efbf-495b-9e7d-0a3b35f71217 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:35.228279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:35.232125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23f0e90-99bd-4054-ab31-3dfcf806c1bb None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:35.237807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ec134d-efbf-495b-9e7d-0a3b35f71217 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:35.275929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2c3be4-7a7f-4f0c-ba24-fcd424c3a4bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:35.276859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2c3be4-7a7f-4f0c-ba24-fcd424c3a4bc None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:35.277139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2c3be4-7a7f-4f0c-ba24-fcd424c3a4bc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:35.277412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2c3be4-7a7f-4f0c-ba24-fcd424c3a4bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:35.278298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:35.279089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:35.279518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:35.279930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:35.281951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] RBAC: Adding query filter params (role.id=974299ae4a414939a4f7e186921d4569, user.id=0ae5216cd2814e7e89183ec170c3b8f8, scope.system=all) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:35.282411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:35.284023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:35.290586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 114/220] 10.222.0.20 () {56 vars in 1308 bytes} [Wed Oct 19 21:42:35 2022] GET /identity/v3/role_assignments?user.id=0ae5216cd2814e7e89183ec170c3b8f8&scope.system=all&role.id=974299ae4a414939a4f7e186921d4569 => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:35.350427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:35.362974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2c3be4-7a7f-4f0c-ba24-fcd424c3a4bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:35.364787 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2c3be4-7a7f-4f0c-ba24-fcd424c3a4bc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:35.477960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 107/221] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 19 21:42:35.484167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:35.594977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afcdec1-731c-43dc-be1f-979274bca48b None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:35.629533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfb929f-5878-4080-b521-16a7f588d9d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:35.630164 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb929f-5878-4080-b521-16a7f588d9d8 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:35.630438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb929f-5878-4080-b521-16a7f588d9d8 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:35.630705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfb929f-5878-4080-b521-16a7f588d9d8 None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:35.631858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb929f-5878-4080-b521-16a7f588d9d8 None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:35.633425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfb929f-5878-4080-b521-16a7f588d9d8 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:35.635030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 115/222] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:35 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:35.641581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2c3be4-7a7f-4f0c-ba24-fcd424c3a4bc None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:35.741819 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:35.751402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2c3be4-7a7f-4f0c-ba24-fcd424c3a4bc None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:35.794152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf21e656-ef69-4b8b-9185-78ba59af73c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:35.794152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf21e656-ef69-4b8b-9185-78ba59af73c5 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:35.794152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf21e656-ef69-4b8b-9185-78ba59af73c5 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:35.794152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf21e656-ef69-4b8b-9185-78ba59af73c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:35.794528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf21e656-ef69-4b8b-9185-78ba59af73c5 None admin] RBAC: Adding query filter params (role.id=2d48ee5817eb4e4388e1a55c7db0b40d, scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18, user.id=0d61dff869894d7897f5ad3461a19e7c) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:35.794766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf21e656-ef69-4b8b-9185-78ba59af73c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:35.796158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf21e656-ef69-4b8b-9185-78ba59af73c5 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:35.802913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 108/223] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:42:35 2022] GET /identity/v3/role_assignments?user.id=0d61dff869894d7897f5ad3461a19e7c&scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18&role.id=2d48ee5817eb4e4388e1a55c7db0b40d => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:37.068476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa888fa-131e-4cd1-ae60-19305f5cfaf2 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.068971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa888fa-131e-4cd1-ae60-19305f5cfaf2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.068971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa888fa-131e-4cd1-ae60-19305f5cfaf2 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.070444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 116/224] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:37.076115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955424b-d327-45a4-9f4f-5f79c68650ee None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.076548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955424b-d327-45a4-9f4f-5f79c68650ee None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.076733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955424b-d327-45a4-9f4f-5f79c68650ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.094861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8955424b-d327-45a4-9f4f-5f79c68650ee None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:37.096069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8955424b-d327-45a4-9f4f-5f79c68650ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:37.154039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549939a5-e738-4a16-96d6-0ea1d17db15a None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.154339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549939a5-e738-4a16-96d6-0ea1d17db15a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.154585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549939a5-e738-4a16-96d6-0ea1d17db15a None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.155958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 117/225] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:42:37.160877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b960b6c-5780-4aad-a99e-fc1377606ab0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.161157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b960b6c-5780-4aad-a99e-fc1377606ab0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.161431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b960b6c-5780-4aad-a99e-fc1377606ab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.165665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 109/226] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:37.180695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5b960b6c-5780-4aad-a99e-fc1377606ab0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:37.181883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5b960b6c-5780-4aad-a99e-fc1377606ab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:37.208020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:37.249906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 118/227] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:37.434391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8955424b-d327-45a4-9f4f-5f79c68650ee None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:37.505877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9b679-980b-487d-8c0b-8be6bedee436 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.506214 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9b679-980b-487d-8c0b-8be6bedee436 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.506499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac9b679-980b-487d-8c0b-8be6bedee436 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.508003 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 119/228] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:42:37.513463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1532b5c6-d3e6-4914-9ac1-1b686db91a8a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.513746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1532b5c6-d3e6-4914-9ac1-1b686db91a8a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.514020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1532b5c6-d3e6-4914-9ac1-1b686db91a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.519471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:37.532561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-1532b5c6-d3e6-4914-9ac1-1b686db91a8a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:37.533746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1532b5c6-d3e6-4914-9ac1-1b686db91a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:37.534070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8955424b-d327-45a4-9f4f-5f79c68650ee None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:37.568556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559dd493-488d-467a-ad20-3423be6cd624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:37.569174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559dd493-488d-467a-ad20-3423be6cd624 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.569448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559dd493-488d-467a-ad20-3423be6cd624 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.569715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559dd493-488d-467a-ad20-3423be6cd624 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.579972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559dd493-488d-467a-ad20-3423be6cd624 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-559dd493-488d-467a-ad20-3423be6cd624 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:37.584711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:37.585554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:37.586321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 110/229] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:37.587007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559dd493-488d-467a-ad20-3423be6cd624 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:37.602121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 120/230] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:37.602977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1532b5c6-d3e6-4914-9ac1-1b686db91a8a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:37.694083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559dd493-488d-467a-ad20-3423be6cd624 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:37.702417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:37.711846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1532b5c6-d3e6-4914-9ac1-1b686db91a8a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:37.728706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c3c48c-cc17-4ffb-b120-ff3d6ebbe096 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:37.729361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c3c48c-cc17-4ffb-b120-ff3d6ebbe096 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.730213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c3c48c-cc17-4ffb-b120-ff3d6ebbe096 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.730489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c3c48c-cc17-4ffb-b120-ff3d6ebbe096 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.731619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c3c48c-cc17-4ffb-b120-ff3d6ebbe096 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:37.733352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c3c48c-cc17-4ffb-b120-ff3d6ebbe096 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:37.735111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 111/231] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:37.740530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c3c48c-cc17-4ffb-b120-ff3d6ebbe096 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:37.747253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:37.747865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.748167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.748504 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.749171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] RBAC: Adding query filter params (name=reader) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:37.749476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:37.750862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:37.758134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:37.763996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 121/232] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:37.765307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:37.855845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c3c48c-cc17-4ffb-b120-ff3d6ebbe096 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:37.870915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-743e3876-16a4-48ab-8a8a-fa8f21807405 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:37.891104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17bcf5c-9b1f-4951-b9ee-0041875c658c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:37.891715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17bcf5c-9b1f-4951-b9ee-0041875c658c None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.891996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17bcf5c-9b1f-4951-b9ee-0041875c658c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.892266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17bcf5c-9b1f-4951-b9ee-0041875c658c None admin] PATH_INFO: `/v3/groups` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.892901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17bcf5c-9b1f-4951-b9ee-0041875c658c None admin] RBAC: Authorizing `identity:create_group()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:37.894351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17bcf5c-9b1f-4951-b9ee-0041875c658c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:37.903889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c509827-1264-4422-b742-1b7721b3b158 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:37.904516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c509827-1264-4422-b742-1b7721b3b158 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:37.904795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c509827-1264-4422-b742-1b7721b3b158 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:37.905065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c509827-1264-4422-b742-1b7721b3b158 None admin] PATH_INFO: `/v3/users/5d3859c317154608b0ff36bfd7b37eeb` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:37.918376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c509827-1264-4422-b742-1b7721b3b158 None admin] RBAC: Authorizing `identity:get_user(user_id=5d3859c317154608b0ff36bfd7b37eeb)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:37.919831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c509827-1264-4422-b742-1b7721b3b158 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:37.921442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 122/233] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:37 2022] GET /identity/v3/users/5d3859c317154608b0ff36bfd7b37eeb => generated 293 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:37.922281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c509827-1264-4422-b742-1b7721b3b158 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:37.936660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:37.965552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 112/234] 10.222.0.20 () {60 vars in 1134 bytes} [Wed Oct 19 21:42:37 2022] POST /identity/v3/groups => generated 229 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:37.967923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17bcf5c-9b1f-4951-b9ee-0041875c658c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:38.031749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c509827-1264-4422-b742-1b7721b3b158 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:38.054119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:38.069987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec1b12-d88f-42e3-9e8d-57781e6450ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:38.070634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec1b12-d88f-42e3-9e8d-57781e6450ea None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:38.071053 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec1b12-d88f-42e3-9e8d-57781e6450ea None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:38.071327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ec1b12-d88f-42e3-9e8d-57781e6450ea None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:38.085329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17bcf5c-9b1f-4951-b9ee-0041875c658c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:38.086920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ec1b12-d88f-42e3-9e8d-57781e6450ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-d4ec1b12-d88f-42e3-9e8d-57781e6450ea None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:38.092541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:38.093512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:38.094286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 123/235] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:38.099822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec1b12-d88f-42e3-9e8d-57781e6450ea None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:38.127424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:38.128049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:38.128322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:38.128595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:38.129337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] RBAC: Adding query filter params (role.id=2d48ee5817eb4e4388e1a55c7db0b40d, user.id=5d3859c317154608b0ff36bfd7b37eeb, scope.system=all) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:38.129643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:38.131132 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:38.138126 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 113/236] 10.222.0.20 () {56 vars in 1308 bytes} [Wed Oct 19 21:42:37 2022] GET /identity/v3/role_assignments?user.id=5d3859c317154608b0ff36bfd7b37eeb&scope.system=all&role.id=2d48ee5817eb4e4388e1a55c7db0b40d => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:38.191581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:38.224347 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ec1b12-d88f-42e3-9e8d-57781e6450ea None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:38.262919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:38.263587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:38.263865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:38.264136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:38.264885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] RBAC: Adding query filter params (name=member) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:38.265328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:38.266816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:38.276675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 124/237] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:38.282479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:38.430710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b02061b-b883-4f48-990e-60364c976f08 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:38.465220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0beb63-01a8-4ad0-a889-70a9a4742eb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:38.465936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0beb63-01a8-4ad0-a889-70a9a4742eb9 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:38.466277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0beb63-01a8-4ad0-a889-70a9a4742eb9 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:38.466611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0beb63-01a8-4ad0-a889-70a9a4742eb9 None admin] PATH_INFO: `/v3/users/0ae5216cd2814e7e89183ec170c3b8f8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:38.467838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0beb63-01a8-4ad0-a889-70a9a4742eb9 None admin] RBAC: Authorizing `identity:get_user(user_id=0ae5216cd2814e7e89183ec170c3b8f8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:38.469411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0beb63-01a8-4ad0-a889-70a9a4742eb9 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:38.471401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 114/238] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:38 2022] GET /identity/v3/users/0ae5216cd2814e7e89183ec170c3b8f8 => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:38.476797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:38.581163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b5aab-3e69-4167-89c7-c27666877017 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:38.616274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3084c435-93a9-41ce-a23c-2450dc0c6c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:38.616917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084c435-93a9-41ce-a23c-2450dc0c6c23 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:38.617190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084c435-93a9-41ce-a23c-2450dc0c6c23 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:38.617462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3084c435-93a9-41ce-a23c-2450dc0c6c23 None admin] PATH_INFO: `/v3/system/users/0ae5216cd2814e7e89183ec170c3b8f8/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:38.618844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3084c435-93a9-41ce-a23c-2450dc0c6c23 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=0ae5216cd2814e7e89183ec170c3b8f8, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:38.620316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3084c435-93a9-41ce-a23c-2450dc0c6c23 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:38.621784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:38.692281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 125/239] 10.222.0.20 () {58 vars in 1334 bytes} [Wed Oct 19 21:42:38 2022] PUT /identity/v3/system/users/0ae5216cd2814e7e89183ec170c3b8f8/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:40.181507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f061b3-0403-4371-a5bf-88ed47d2c372 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.182019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f061b3-0403-4371-a5bf-88ed47d2c372 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.182019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f061b3-0403-4371-a5bf-88ed47d2c372 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.183441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 115/240] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:40.189324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a03bc9-a02d-4772-95f0-cc2a8a3b0251 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.189838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a03bc9-a02d-4772-95f0-cc2a8a3b0251 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.189910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a03bc9-a02d-4772-95f0-cc2a8a3b0251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.210786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-14a03bc9-a02d-4772-95f0-cc2a8a3b0251 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:40.212005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-14a03bc9-a02d-4772-95f0-cc2a8a3b0251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:40.279293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 126/241] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:40.287654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4d289f-bdad-4e48-a7b5-2147aeac42fd None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.287941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4d289f-bdad-4e48-a7b5-2147aeac42fd None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.288221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4d289f-bdad-4e48-a7b5-2147aeac42fd None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.289565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 116/242] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:40.295270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d60c8c-cbf8-4aec-9ddf-b78dd102868c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.295587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d60c8c-cbf8-4aec-9ddf-b78dd102868c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.295867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d60c8c-cbf8-4aec-9ddf-b78dd102868c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.314215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-24d60c8c-cbf8-4aec-9ddf-b78dd102868c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:40.315423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-24d60c8c-cbf8-4aec-9ddf-b78dd102868c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:40.387607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 127/243] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:40.551213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4d289f-bdad-4e48-a7b5-2147aeac42fd None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:40.561272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:40.655343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4d289f-bdad-4e48-a7b5-2147aeac42fd None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:40.658469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d60c8c-cbf8-4aec-9ddf-b78dd102868c None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:40.691679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c128d9db-d967-4728-80ee-a288ced0da70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:40.692311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c128d9db-d967-4728-80ee-a288ced0da70 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.692582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c128d9db-d967-4728-80ee-a288ced0da70 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.692850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c128d9db-d967-4728-80ee-a288ced0da70 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.693946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c128d9db-d967-4728-80ee-a288ced0da70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:40.695528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c128d9db-d967-4728-80ee-a288ced0da70 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:40.697287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 117/244] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:40.702569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c128d9db-d967-4728-80ee-a288ced0da70 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:40.719224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:40.767324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d60c8c-cbf8-4aec-9ddf-b78dd102868c None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:40.784895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:40.817140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb8871a-7218-4dbd-bf25-2fe91421fd23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:40.817498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c128d9db-d967-4728-80ee-a288ced0da70 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:40.817700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb8871a-7218-4dbd-bf25-2fe91421fd23 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.817981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb8871a-7218-4dbd-bf25-2fe91421fd23 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.818251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb8871a-7218-4dbd-bf25-2fe91421fd23 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.831781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb8871a-7218-4dbd-bf25-2fe91421fd23 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-5bb8871a-7218-4dbd-bf25-2fe91421fd23 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:40.836808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:40.837702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:40.838848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 128/245] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:40 2022] GET /identity/v3/roles/reader => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:40.839821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e9b74-fdca-48d3-8fff-66315369c67e None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.839821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e9b74-fdca-48d3-8fff-66315369c67e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.840297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f5e9b74-fdca-48d3-8fff-66315369c67e None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.841541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 129/246] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:40.844175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5e9b74-fdca-48d3-8fff-66315369c67e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:40.857029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8626d673-e15e-4f58-ba68-259aa822ddf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:40.857642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8626d673-e15e-4f58-ba68-259aa822ddf1 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.857924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8626d673-e15e-4f58-ba68-259aa822ddf1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.858200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8626d673-e15e-4f58-ba68-259aa822ddf1 None admin] PATH_INFO: `/v3/groups` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.858839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8626d673-e15e-4f58-ba68-259aa822ddf1 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:40.860327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8626d673-e15e-4f58-ba68-259aa822ddf1 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:40.937049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 118/247] 10.222.0.20 () {60 vars in 1134 bytes} [Wed Oct 19 21:42:40 2022] POST /identity/v3/groups => generated 226 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:42:40.942258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f5e9b74-fdca-48d3-8fff-66315369c67e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:40.944500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc1074-8488-4799-a313-4c63d9ed1f18 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.944925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc1074-8488-4799-a313-4c63d9ed1f18 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.945258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dc1074-8488-4799-a313-4c63d9ed1f18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.963102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:40.965064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-27dc1074-8488-4799-a313-4c63d9ed1f18 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:40.966397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-27dc1074-8488-4799-a313-4c63d9ed1f18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:40.985958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:40.986666 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:40.986943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:40.987235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:40.987917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] RBAC: Adding query filter params (name=reader) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:40.988218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:40.989645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:41.000044 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 130/248] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:40 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:41.005627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:41.045331 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 119/249] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:41.118306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2cec14-dc0d-443a-8715-1d3a649b3a35 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:41.153318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1857e068-9ddb-4ad9-894e-5eb2b29673fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:41.153952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1857e068-9ddb-4ad9-894e-5eb2b29673fc None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:41.154230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1857e068-9ddb-4ad9-894e-5eb2b29673fc None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:41.154501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1857e068-9ddb-4ad9-894e-5eb2b29673fc None admin] PATH_INFO: `/v3/users/5d3859c317154608b0ff36bfd7b37eeb` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:41.155738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1857e068-9ddb-4ad9-894e-5eb2b29673fc None admin] RBAC: Authorizing `identity:get_user(user_id=5d3859c317154608b0ff36bfd7b37eeb)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:41.157194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1857e068-9ddb-4ad9-894e-5eb2b29673fc None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:41.158833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 131/250] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:41 2022] GET /identity/v3/users/5d3859c317154608b0ff36bfd7b37eeb => generated 293 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:41.164694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dc1074-8488-4799-a313-4c63d9ed1f18 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:41.232610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:41.271686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dc1074-8488-4799-a313-4c63d9ed1f18 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:41.307422 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d30308-7baf-43be-b93e-6475a7d5628b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:41.308625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d30308-7baf-43be-b93e-6475a7d5628b None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:41.309125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d30308-7baf-43be-b93e-6475a7d5628b None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:41.309468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d30308-7baf-43be-b93e-6475a7d5628b None admin] PATH_INFO: `/v3/system/users/5d3859c317154608b0ff36bfd7b37eeb/roles/2d48ee5817eb4e4388e1a55c7db0b40d` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:41.311124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d30308-7baf-43be-b93e-6475a7d5628b None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=5d3859c317154608b0ff36bfd7b37eeb, role_id=2d48ee5817eb4e4388e1a55c7db0b40d)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:41.312759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d30308-7baf-43be-b93e-6475a7d5628b None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:41.318888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1857e068-9ddb-4ad9-894e-5eb2b29673fc None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:41.372443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:41.378656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 120/251] 10.222.0.20 () {58 vars in 1334 bytes} [Wed Oct 19 21:42:41 2022] PUT /identity/v3/system/users/5d3859c317154608b0ff36bfd7b37eeb/roles/2d48ee5817eb4e4388e1a55c7db0b40d => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:41.441576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1857e068-9ddb-4ad9-894e-5eb2b29673fc None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:41.486787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cf9326-8ee6-4f87-9555-0e6f620a1db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:41.487783 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cf9326-8ee6-4f87-9555-0e6f620a1db4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:41.488201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cf9326-8ee6-4f87-9555-0e6f620a1db4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:41.488612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cf9326-8ee6-4f87-9555-0e6f620a1db4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:41.501047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cf9326-8ee6-4f87-9555-0e6f620a1db4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-a7cf9326-8ee6-4f87-9555-0e6f620a1db4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:41.507013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:41.508134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:41.509096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:41.512042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 132/252] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:41.518385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d30308-7baf-43be-b93e-6475a7d5628b None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:41.638796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d30308-7baf-43be-b93e-6475a7d5628b None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:41.667601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:41.675450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:41.676058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:41.676334 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:41.676606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:41.677447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] RBAC: Adding query filter params (name=member) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:41.677818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:41.679336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:41.689479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 121/253] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:41.695261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cf9326-8ee6-4f87-9555-0e6f620a1db4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:41.824738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cf9326-8ee6-4f87-9555-0e6f620a1db4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:41.837660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:41.865368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2f90f7-2472-460a-91b5-82759d84ed2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:41.865999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f90f7-2472-460a-91b5-82759d84ed2b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:41.866275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f90f7-2472-460a-91b5-82759d84ed2b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:41.866544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f90f7-2472-460a-91b5-82759d84ed2b None admin] PATH_INFO: `/v3/users/0ae5216cd2814e7e89183ec170c3b8f8` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:41.867762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2f90f7-2472-460a-91b5-82759d84ed2b None admin] RBAC: Authorizing `identity:get_user(user_id=0ae5216cd2814e7e89183ec170c3b8f8)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:41.869216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2f90f7-2472-460a-91b5-82759d84ed2b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:41.871387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 133/254] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:41 2022] GET /identity/v3/users/0ae5216cd2814e7e89183ec170c3b8f8 => generated 293 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:41.877426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:42.001898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8a76e7-6004-4b4d-976f-fd6d7e651691 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:42.048254 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503ea87b-55d9-457e-849f-db02f4d11c44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:42.048863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503ea87b-55d9-457e-849f-db02f4d11c44 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:42.049142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503ea87b-55d9-457e-849f-db02f4d11c44 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:42.049415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503ea87b-55d9-457e-849f-db02f4d11c44 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:42.050159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503ea87b-55d9-457e-849f-db02f4d11c44 None admin] RBAC: Adding query filter params (role.id=974299ae4a414939a4f7e186921d4569, user.id=0ae5216cd2814e7e89183ec170c3b8f8, scope.system=all) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:42.050477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503ea87b-55d9-457e-849f-db02f4d11c44 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:42.051963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503ea87b-55d9-457e-849f-db02f4d11c44 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:42.066264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 122/255] 10.222.0.20 () {56 vars in 1308 bytes} [Wed Oct 19 21:42:41 2022] GET /identity/v3/role_assignments?user.id=0ae5216cd2814e7e89183ec170c3b8f8&scope.system=all&role.id=974299ae4a414939a4f7e186921d4569 => generated 510 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:43.611230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057cb271-19f9-46a6-8137-9552222c0c5d None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:43.611742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057cb271-19f9-46a6-8137-9552222c0c5d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:43.611742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057cb271-19f9-46a6-8137-9552222c0c5d None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:43.613366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 134/256] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:43.624453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf2181b-adfa-4d4b-9781-7ac7badc0051 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:43.624906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf2181b-adfa-4d4b-9781-7ac7badc0051 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:43.625008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf2181b-adfa-4d4b-9781-7ac7badc0051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:43.652725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-dbf2181b-adfa-4d4b-9781-7ac7badc0051 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:43.653925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dbf2181b-adfa-4d4b-9781-7ac7badc0051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:43.707171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:43.719995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 123/257] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:43 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 21:42:44.062030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919c41d-8b9c-48f5-985c-e9e9939f6529 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.062302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919c41d-8b9c-48f5-985c-e9e9939f6529 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.062580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919c41d-8b9c-48f5-985c-e9e9939f6529 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.064268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 135/258] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:44.071341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c8909-34b6-4b8d-9fe9-5301322f37b5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.071624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c8909-34b6-4b8d-9fe9-5301322f37b5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.071892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6c8909-34b6-4b8d-9fe9-5301322f37b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.095627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5e6c8909-34b6-4b8d-9fe9-5301322f37b5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:44.096850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5e6c8909-34b6-4b8d-9fe9-5301322f37b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:44.110624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 124/259] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:44 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 21:42:44.288399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919c41d-8b9c-48f5-985c-e9e9939f6529 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:44.303078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919c41d-8b9c-48f5-985c-e9e9939f6529 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:44.309858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ac6c0-a6f1-4ccb-b50c-5f9c02749d3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:44.310498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ac6c0-a6f1-4ccb-b50c-5f9c02749d3b admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.310813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ac6c0-a6f1-4ccb-b50c-5f9c02749d3b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.311107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661ac6c0-a6f1-4ccb-b50c-5f9c02749d3b admin admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.319191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a7c1ca-d368-4f35-a9c1-d489e733d738 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.319442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a7c1ca-d368-4f35-a9c1-d489e733d738 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.319669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a7c1ca-d368-4f35-a9c1-d489e733d738 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.321013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 125/260] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:42:44.326917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfae413-0dc8-4823-baa8-6bfdeaf6e8dc None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.327226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfae413-0dc8-4823-baa8-6bfdeaf6e8dc None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.327557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcfae413-0dc8-4823-baa8-6bfdeaf6e8dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.349042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661ac6c0-a6f1-4ccb-b50c-5f9c02749d3b admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-661ac6c0-a6f1-4ccb-b50c-5f9c02749d3b admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:44.362254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:44.363195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:44.363979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 136/261] 10.222.0.20 () {56 vars in 1120 bytes} [Wed Oct 19 21:42:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 76 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:44.364610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-bcfae413-0dc8-4823-baa8-6bfdeaf6e8dc None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:44.365409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-bcfae413-0dc8-4823-baa8-6bfdeaf6e8dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:44.387495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ac6c0-a6f1-4ccb-b50c-5f9c02749d3b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:44.399192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661ac6c0-a6f1-4ccb-b50c-5f9c02749d3b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:44.411890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:44.412556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.412822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.413086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.413761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] RBAC: Adding query filter params (name=member) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:44.414056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:44.419669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING py.warnings [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:42:44.419669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: warnings.warn(msg) Oct 19 21:42:44.419669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: Oct 19 21:42:44.423383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:44.452924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 137/262] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:42:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:44.476104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:44.487413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3be434-934d-4c4e-b765-4eb961924007 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:44.496521 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:44.498417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.498699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.498979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] PATH_INFO: `/v3/groups/4959a820d03549248dc8bed9f38d3012` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.513684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] RBAC: Authorizing `identity:get_group(group_id=4959a820d03549248dc8bed9f38d3012)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:44.514735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING py.warnings [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:42:44.514735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: warnings.warn(msg) Oct 19 21:42:44.514735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: Oct 19 21:42:44.519360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:44.521087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 138/263] 10.222.0.20 () {56 vars in 1201 bytes} [Wed Oct 19 21:42:44 2022] GET /identity/v3/groups/4959a820d03549248dc8bed9f38d3012 => generated 226 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:44.528266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:44.532672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 126/264] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:44.541387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81aff375-4c84-410b-9067-e7445b46f3ca admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:44.550816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b848d8-fc08-4238-a9d4-3ef542a97668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:44.551473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b848d8-fc08-4238-a9d4-3ef542a97668 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.551750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b848d8-fc08-4238-a9d4-3ef542a97668 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.552019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b848d8-fc08-4238-a9d4-3ef542a97668 admin admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.553506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b848d8-fc08-4238-a9d4-3ef542a97668 admin admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:44.555630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b848d8-fc08-4238-a9d4-3ef542a97668 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:44.557649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 139/265] 10.222.0.20 () {56 vars in 1207 bytes} [Wed Oct 19 21:42:44 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:44.563828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfae413-0dc8-4823-baa8-6bfdeaf6e8dc None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:44.584808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcfae413-0dc8-4823-baa8-6bfdeaf6e8dc None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:44.584808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:44.585237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.585509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.585968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df/groups/4959a820d03549248dc8bed9f38d3012/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.588014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebd577f63e0b41edb25465e82d9af4df, group_id=4959a820d03549248dc8bed9f38d3012, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:44.589857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:42:44.589857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 21:42:44.589857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 21:42:44.596212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:44.697922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 127/266] 10.222.0.20 () {58 vars in 1463 bytes} [Wed Oct 19 21:42:44 2022] PUT /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df/groups/4959a820d03549248dc8bed9f38d3012/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:44.772864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b848d8-fc08-4238-a9d4-3ef542a97668 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:44.788221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b848d8-fc08-4238-a9d4-3ef542a97668 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:44.885899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da66441e-4501-479b-ac51-0e0929e3f9f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:44.886847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66441e-4501-479b-ac51-0e0929e3f9f2 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.887293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66441e-4501-479b-ac51-0e0929e3f9f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.887690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da66441e-4501-479b-ac51-0e0929e3f9f2 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.899635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da66441e-4501-479b-ac51-0e0929e3f9f2 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-da66441e-4501-479b-ac51-0e0929e3f9f2 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:44.905163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:44.906067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:44.906834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 140/267] 10.222.0.20 () {56 vars in 1135 bytes} [Wed Oct 19 21:42:44 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 134 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:44.912467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:44.933614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5282184-4fcc-4e6e-b883-d16842c7dca2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:44.940047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da66441e-4501-479b-ac51-0e0929e3f9f2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:44.943611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b93e21-f7b2-4d8b-9ffe-0971f6ee2692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:44.944564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b93e21-f7b2-4d8b-9ffe-0971f6ee2692 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.945113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b93e21-f7b2-4d8b-9ffe-0971f6ee2692 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.945592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b93e21-f7b2-4d8b-9ffe-0971f6ee2692 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.946476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b93e21-f7b2-4d8b-9ffe-0971f6ee2692 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:44.947016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b93e21-f7b2-4d8b-9ffe-0971f6ee2692 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:44.948293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-b4b93e21-f7b2-4d8b-9ffe-0971f6ee2692 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:42:44.948293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 21:42:44.948293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 21:42:44.949534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b93e21-f7b2-4d8b-9ffe-0971f6ee2692 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:44.967347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 128/268] 10.222.0.20 () {56 vars in 1132 bytes} [Wed Oct 19 21:42:44 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:44.975519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eda51e-542f-423a-8b8b-96505acaa7e0 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.976195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eda51e-542f-423a-8b8b-96505acaa7e0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.976671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6eda51e-542f-423a-8b8b-96505acaa7e0 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:44.978358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 129/269] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:42:44.991018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3283abc7-565f-44c2-85b6-6b3f5500e2a9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:44.991631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3283abc7-565f-44c2-85b6-6b3f5500e2a9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:44.992196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3283abc7-565f-44c2-85b6-6b3f5500e2a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.032217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3283abc7-565f-44c2-85b6-6b3f5500e2a9 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:45.033884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3283abc7-565f-44c2-85b6-6b3f5500e2a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:45.044792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 130/270] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:44 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 19 21:42:45.045727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3283abc7-565f-44c2-85b6-6b3f5500e2a9 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.051231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:45.057256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3283abc7-565f-44c2-85b6-6b3f5500e2a9 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.063818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.064515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.064791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.065104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] PATH_INFO: `/v3/groups/4959a820d03549248dc8bed9f38d3012` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.066404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] RBAC: Authorizing `identity:get_group(group_id=4959a820d03549248dc8bed9f38d3012)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.067509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:get_group": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.group.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:42:45.067509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 21:42:45.067509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 21:42:45.068762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.070556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 131/271] 10.222.0.20 () {56 vars in 1201 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/groups/4959a820d03549248dc8bed9f38d3012 => generated 226 bytes in 25 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:45.076959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.083734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da66441e-4501-479b-ac51-0e0929e3f9f2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.088173 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33674f15-577f-431f-9ffc-f1b0a2863bcc admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.093642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e112c54-22a9-4a56-bd10-d6acb37a5c8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.094271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e112c54-22a9-4a56-bd10-d6acb37a5c8b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.094551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e112c54-22a9-4a56-bd10-d6acb37a5c8b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.094872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e112c54-22a9-4a56-bd10-d6acb37a5c8b admin admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.100790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e112c54-22a9-4a56-bd10-d6acb37a5c8b admin admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.106330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e112c54-22a9-4a56-bd10-d6acb37a5c8b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.108544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 132/272] 10.222.0.20 () {56 vars in 1207 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:45.114403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e112c54-22a9-4a56-bd10-d6acb37a5c8b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.126839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e112c54-22a9-4a56-bd10-d6acb37a5c8b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.138119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901ed4a6-38fe-49c1-bb7a-81edef2687ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.138730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901ed4a6-38fe-49c1-bb7a-81edef2687ea admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.139231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901ed4a6-38fe-49c1-bb7a-81edef2687ea admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.139508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901ed4a6-38fe-49c1-bb7a-81edef2687ea admin admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df/groups/4959a820d03549248dc8bed9f38d3012/roles/3cb31ad012324a57acd3a4bd258d1fb1` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.145416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a319e94-8c95-42ca-8632-ac5bb026aaa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.146167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901ed4a6-38fe-49c1-bb7a-81edef2687ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=ebd577f63e0b41edb25465e82d9af4df, group_id=4959a820d03549248dc8bed9f38d3012, role_id=3cb31ad012324a57acd3a4bd258d1fb1)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.148240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a319e94-8c95-42ca-8632-ac5bb026aaa0 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.148519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a319e94-8c95-42ca-8632-ac5bb026aaa0 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.148786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a319e94-8c95-42ca-8632-ac5bb026aaa0 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.151927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901ed4a6-38fe-49c1-bb7a-81edef2687ea admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.165543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a319e94-8c95-42ca-8632-ac5bb026aaa0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-7a319e94-8c95-42ca-8632-ac5bb026aaa0 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:45.171519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:45.172378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:45.173162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 141/273] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:44 2022] GET /identity/v3/roles/reader => generated 84 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:45.178724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a319e94-8c95-42ca-8632-ac5bb026aaa0 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.223304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 133/274] 10.222.0.20 () {58 vars in 1463 bytes} [Wed Oct 19 21:42:45 2022] PUT /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df/groups/4959a820d03549248dc8bed9f38d3012/roles/3cb31ad012324a57acd3a4bd258d1fb1 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:45.224737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf1b083-d2cc-4f45-9272-190e4c4c5861 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.225077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf1b083-d2cc-4f45-9272-190e4c4c5861 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.225419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf1b083-d2cc-4f45-9272-190e4c4c5861 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.226864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 134/275] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:45.233099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec296f3a-8788-45e7-83ac-f3f05272e7c2 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.233404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec296f3a-8788-45e7-83ac-f3f05272e7c2 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.233659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec296f3a-8788-45e7-83ac-f3f05272e7c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.264481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ec296f3a-8788-45e7-83ac-f3f05272e7c2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:45.265716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ec296f3a-8788-45e7-83ac-f3f05272e7c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:45.344281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 135/276] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:45 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 21:42:45.376598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a319e94-8c95-42ca-8632-ac5bb026aaa0 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.418054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec296f3a-8788-45e7-83ac-f3f05272e7c2 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.429186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.429803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.429912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec296f3a-8788-45e7-83ac-f3f05272e7c2 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.430076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.430333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.431031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] RBAC: Adding query filter params (name=reader) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:45.431332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.433348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.437252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8449f1e8-25c7-42f8-bdf5-d22a779241a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.437902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8449f1e8-25c7-42f8-bdf5-d22a779241a7 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.438177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8449f1e8-25c7-42f8-bdf5-d22a779241a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.438447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8449f1e8-25c7-42f8-bdf5-d22a779241a7 admin admin] PATH_INFO: `/v3/roles/member` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.450493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 142/277] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:45.454125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8449f1e8-25c7-42f8-bdf5-d22a779241a7 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:45.455992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-8449f1e8-25c7-42f8-bdf5-d22a779241a7 admin admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:45.459822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:45.460707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:45.461476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 136/278] 10.222.0.20 () {56 vars in 1120 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/roles/member => generated 84 bytes in 45 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:45.462918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:45.472917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8449f1e8-25c7-42f8-bdf5-d22a779241a7 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.486494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8449f1e8-25c7-42f8-bdf5-d22a779241a7 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.491263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:45.497546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.498177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.498451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.498722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.499446 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] RBAC: Adding query filter params (name=member) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:45.499753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.501425 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.522595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 137/279] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:45.529081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.549247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c483d29-8dd0-492e-82b3-eeff74c02cec admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.562489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb183516-9e7c-4752-927f-79b4c40e39b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.563165 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb183516-9e7c-4752-927f-79b4c40e39b6 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.563443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb183516-9e7c-4752-927f-79b4c40e39b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.563713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb183516-9e7c-4752-927f-79b4c40e39b6 admin admin] PATH_INFO: `/v3/groups/4959a820d03549248dc8bed9f38d3012` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.564842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb183516-9e7c-4752-927f-79b4c40e39b6 admin admin] RBAC: Authorizing `identity:get_group(group_id=4959a820d03549248dc8bed9f38d3012)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.566766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb183516-9e7c-4752-927f-79b4c40e39b6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.574621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 138/280] 10.222.0.20 () {56 vars in 1201 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/groups/4959a820d03549248dc8bed9f38d3012 => generated 226 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:45.575564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb183516-9e7c-4752-927f-79b4c40e39b6 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.599430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb183516-9e7c-4752-927f-79b4c40e39b6 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.615142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5202c030-52ed-4b74-a3a7-e906645b1f75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.615790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5202c030-52ed-4b74-a3a7-e906645b1f75 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.616072 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5202c030-52ed-4b74-a3a7-e906645b1f75 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.616344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5202c030-52ed-4b74-a3a7-e906645b1f75 admin admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.617495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5202c030-52ed-4b74-a3a7-e906645b1f75 admin admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.619730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5202c030-52ed-4b74-a3a7-e906645b1f75 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.622274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 139/281] 10.222.0.20 () {56 vars in 1207 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:45.630921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5202c030-52ed-4b74-a3a7-e906645b1f75 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.646944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5202c030-52ed-4b74-a3a7-e906645b1f75 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.663335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fd3624-980e-443b-a9c6-c36ae910da27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.663997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd3624-980e-443b-a9c6-c36ae910da27 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.664270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd3624-980e-443b-a9c6-c36ae910da27 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.664539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd3624-980e-443b-a9c6-c36ae910da27 admin admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/groups/4959a820d03549248dc8bed9f38d3012/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.668252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fd3624-980e-443b-a9c6-c36ae910da27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18, group_id=4959a820d03549248dc8bed9f38d3012, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.670847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fd3624-980e-443b-a9c6-c36ae910da27 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.678157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c9be6d-3c04-4cd7-9976-06c757334628 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.726470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05ec43-401d-464f-87c7-0e9bb5fb53a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.727195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05ec43-401d-464f-87c7-0e9bb5fb53a4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.727526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05ec43-401d-464f-87c7-0e9bb5fb53a4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.727846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f05ec43-401d-464f-87c7-0e9bb5fb53a4 None admin] PATH_INFO: `/v3/users/5d3859c317154608b0ff36bfd7b37eeb` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.728967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f05ec43-401d-464f-87c7-0e9bb5fb53a4 None admin] RBAC: Authorizing `identity:get_user(user_id=5d3859c317154608b0ff36bfd7b37eeb)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.730631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f05ec43-401d-464f-87c7-0e9bb5fb53a4 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.732606 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 143/282] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/users/5d3859c317154608b0ff36bfd7b37eeb => generated 293 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:45.733428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05ec43-401d-464f-87c7-0e9bb5fb53a4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.747451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f05ec43-401d-464f-87c7-0e9bb5fb53a4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.752592 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a42a1b-cf9b-4840-9cb8-6e9a509849c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.753260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a42a1b-cf9b-4840-9cb8-6e9a509849c4 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.753584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a42a1b-cf9b-4840-9cb8-6e9a509849c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.753856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a42a1b-cf9b-4840-9cb8-6e9a509849c4 admin admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.759248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 140/283] 10.222.0.20 () {58 vars in 1463 bytes} [Wed Oct 19 21:42:45 2022] PUT /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/groups/4959a820d03549248dc8bed9f38d3012/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 21:42:45.760093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fd3624-980e-443b-a9c6-c36ae910da27 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.769048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a42a1b-cf9b-4840-9cb8-6e9a509849c4 admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-67a42a1b-cf9b-4840-9cb8-6e9a509849c4 admin admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:45.774127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:45.777242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:45.779273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 144/284] 10.222.0.20 () {56 vars in 1135 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 42 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:45.790030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a42a1b-cf9b-4840-9cb8-6e9a509849c4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.793199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a42a1b-cf9b-4840-9cb8-6e9a509849c4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.889532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.890141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.890417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.890733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.896140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:45.896445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.898149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.907770 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 145/285] 10.222.0.20 () {56 vars in 1132 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:45.923271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.929419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:45.937603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93f9727b-1033-438a-b9f5-961bb5e2c377 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.942377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c4c408-e175-4d7a-8411-e9b26b2ab2f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.947233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4c408-e175-4d7a-8411-e9b26b2ab2f9 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.951988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4c408-e175-4d7a-8411-e9b26b2ab2f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.951988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c4c408-e175-4d7a-8411-e9b26b2ab2f9 admin admin] PATH_INFO: `/v3/groups/4959a820d03549248dc8bed9f38d3012` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.951988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c4c408-e175-4d7a-8411-e9b26b2ab2f9 admin admin] RBAC: Authorizing `identity:get_group(group_id=4959a820d03549248dc8bed9f38d3012)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.952219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c4c408-e175-4d7a-8411-e9b26b2ab2f9 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.954167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 146/286] 10.222.0.20 () {56 vars in 1201 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/groups/4959a820d03549248dc8bed9f38d3012 => generated 226 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:45.961032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c4c408-e175-4d7a-8411-e9b26b2ab2f9 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:45.971816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c4c408-e175-4d7a-8411-e9b26b2ab2f9 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:45.981271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42a02f7-77b0-471a-952a-a8aaf336313a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:45.982020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42a02f7-77b0-471a-952a-a8aaf336313a admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:45.982241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42a02f7-77b0-471a-952a-a8aaf336313a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:45.982514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42a02f7-77b0-471a-952a-a8aaf336313a admin admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:45.983741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42a02f7-77b0-471a-952a-a8aaf336313a admin admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:45.986428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42a02f7-77b0-471a-952a-a8aaf336313a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:45.988159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 147/287] 10.222.0.20 () {56 vars in 1207 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:46.019691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42a02f7-77b0-471a-952a-a8aaf336313a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:46.019880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fd3624-980e-443b-a9c6-c36ae910da27 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:46.043130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42a02f7-77b0-471a-952a-a8aaf336313a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:46.059409 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c3d3fa-16d8-4561-9bc3-39cc846dd560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:46.060094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c3d3fa-16d8-4561-9bc3-39cc846dd560 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:46.060374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c3d3fa-16d8-4561-9bc3-39cc846dd560 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:46.060645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c3d3fa-16d8-4561-9bc3-39cc846dd560 admin admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/groups/4959a820d03549248dc8bed9f38d3012/roles/3cb31ad012324a57acd3a4bd258d1fb1` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:46.064604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c3d3fa-16d8-4561-9bc3-39cc846dd560 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18, group_id=4959a820d03549248dc8bed9f38d3012, role_id=3cb31ad012324a57acd3a4bd258d1fb1)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:46.065664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING py.warnings [None req-e6c3d3fa-16d8-4561-9bc3-39cc846dd560 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:42:46.065664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: warnings.warn(msg) Oct 19 21:42:46.065664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: Oct 19 21:42:46.067663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c3d3fa-16d8-4561-9bc3-39cc846dd560 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:46.092400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a8a46b-3688-46f7-a39a-7e9e96cb7ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:46.093020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a8a46b-3688-46f7-a39a-7e9e96cb7ee3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:46.093298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a8a46b-3688-46f7-a39a-7e9e96cb7ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:46.093638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a8a46b-3688-46f7-a39a-7e9e96cb7ee3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:46.094313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a8a46b-3688-46f7-a39a-7e9e96cb7ee3 None admin] RBAC: Adding query filter params (role.id=2d48ee5817eb4e4388e1a55c7db0b40d, user.id=5d3859c317154608b0ff36bfd7b37eeb, scope.system=all) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:46.101016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a8a46b-3688-46f7-a39a-7e9e96cb7ee3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:46.101016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a8a46b-3688-46f7-a39a-7e9e96cb7ee3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:46.113768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 141/288] 10.222.0.20 () {56 vars in 1308 bytes} [Wed Oct 19 21:42:45 2022] GET /identity/v3/role_assignments?user.id=5d3859c317154608b0ff36bfd7b37eeb&scope.system=all&role.id=2d48ee5817eb4e4388e1a55c7db0b40d => generated 510 bytes in 354 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:46.155872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 148/289] 10.222.0.20 () {58 vars in 1463 bytes} [Wed Oct 19 21:42:46 2022] PUT /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18/groups/4959a820d03549248dc8bed9f38d3012/roles/3cb31ad012324a57acd3a4bd258d1fb1 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:46.621322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4003e-a048-4c67-995e-bc327cd14fe4 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:46.621615 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4003e-a048-4c67-995e-bc327cd14fe4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:46.621853 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e4003e-a048-4c67-995e-bc327cd14fe4 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:46.623296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 142/290] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:46.628518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0542d4-517e-4bcc-8dc7-ef5735077091 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:46.628801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0542d4-517e-4bcc-8dc7-ef5735077091 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:46.629070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0542d4-517e-4bcc-8dc7-ef5735077091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:46.654154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2a0542d4-517e-4bcc-8dc7-ef5735077091 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:46.655392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2a0542d4-517e-4bcc-8dc7-ef5735077091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:46.731324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 149/291] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:46 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 19 21:42:47.155848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e4003e-a048-4c67-995e-bc327cd14fe4 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:47.171394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e4003e-a048-4c67-995e-bc327cd14fe4 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:47.178394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a05408-c55c-413e-b7e9-1ea89629bd4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:47.179034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a05408-c55c-413e-b7e9-1ea89629bd4f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:47.179313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a05408-c55c-413e-b7e9-1ea89629bd4f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:47.179583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a05408-c55c-413e-b7e9-1ea89629bd4f admin admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:47.195008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a05408-c55c-413e-b7e9-1ea89629bd4f admin admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c2a05408-c55c-413e-b7e9-1ea89629bd4f admin admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:47.199856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:47.200713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:47.201487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 143/292] 10.222.0.20 () {56 vars in 1117 bytes} [Wed Oct 19 21:42:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 46 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:47.206743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0542d4-517e-4bcc-8dc7-ef5735077091 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:47.220590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0542d4-517e-4bcc-8dc7-ef5735077091 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:47.231813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:47.232737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:47.233217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:47.233713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:47.234587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] RBAC: Adding query filter params (name=admin) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:47.235133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:47.236951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:47.249813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 150/293] 10.222.0.20 () {56 vars in 1120 bytes} [Wed Oct 19 21:42:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:47.255157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a05408-c55c-413e-b7e9-1ea89629bd4f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:47.268084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a05408-c55c-413e-b7e9-1ea89629bd4f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:47.278247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7edfce-7b80-4f50-a066-8269af63e6fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:47.278876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7edfce-7b80-4f50-a066-8269af63e6fa admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:47.279202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7edfce-7b80-4f50-a066-8269af63e6fa admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:47.279474 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7edfce-7b80-4f50-a066-8269af63e6fa admin admin] PATH_INFO: `/v3/groups/6e52655036924ac5a48cde7936bc03c8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:47.286153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7edfce-7b80-4f50-a066-8269af63e6fa admin admin] RBAC: Authorizing `identity:get_group(group_id=6e52655036924ac5a48cde7936bc03c8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:47.288113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7edfce-7b80-4f50-a066-8269af63e6fa admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:47.289687 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 144/294] 10.222.0.20 () {56 vars in 1201 bytes} [Wed Oct 19 21:42:47 2022] GET /identity/v3/groups/6e52655036924ac5a48cde7936bc03c8 => generated 229 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:47.295709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:47.312087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd66d35-c40f-4ef0-b946-1c551343d207 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:47.323061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60744543-c2b4-4fff-a500-8f343e8d9e1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:47.323989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60744543-c2b4-4fff-a500-8f343e8d9e1e admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:47.324489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60744543-c2b4-4fff-a500-8f343e8d9e1e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:47.324964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60744543-c2b4-4fff-a500-8f343e8d9e1e admin admin] PATH_INFO: `/v3/projects/0fd8fb8e58c74c8cbe244194ce71b304` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:47.331658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60744543-c2b4-4fff-a500-8f343e8d9e1e admin admin] RBAC: Authorizing `identity:get_project(project_id=0fd8fb8e58c74c8cbe244194ce71b304)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:47.333728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60744543-c2b4-4fff-a500-8f343e8d9e1e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:47.335638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 151/295] 10.222.0.20 () {56 vars in 1207 bytes} [Wed Oct 19 21:42:47 2022] GET /identity/v3/projects/0fd8fb8e58c74c8cbe244194ce71b304 => generated 344 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:47.341975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7edfce-7b80-4f50-a066-8269af63e6fa admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:47.356815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7edfce-7b80-4f50-a066-8269af63e6fa admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:47.362901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f365b22b-ea61-4eba-9b45-d5a00f7a3f25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:47.363620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b22b-ea61-4eba-9b45-d5a00f7a3f25 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:47.363897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b22b-ea61-4eba-9b45-d5a00f7a3f25 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:47.364222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f365b22b-ea61-4eba-9b45-d5a00f7a3f25 admin admin] PATH_INFO: `/v3/projects/0fd8fb8e58c74c8cbe244194ce71b304/groups/6e52655036924ac5a48cde7936bc03c8/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:47.366303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365b22b-ea61-4eba-9b45-d5a00f7a3f25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0fd8fb8e58c74c8cbe244194ce71b304, group_id=6e52655036924ac5a48cde7936bc03c8, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:47.369061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f365b22b-ea61-4eba-9b45-d5a00f7a3f25 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:47.424945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:47.429902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 145/296] 10.222.0.20 () {58 vars in 1463 bytes} [Wed Oct 19 21:42:47 2022] PUT /identity/v3/projects/0fd8fb8e58c74c8cbe244194ce71b304/groups/6e52655036924ac5a48cde7936bc03c8/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:42:48.247834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767f04c8-935b-4d4e-80b2-64dd473ac41d None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:48.248911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767f04c8-935b-4d4e-80b2-64dd473ac41d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:48.249402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767f04c8-935b-4d4e-80b2-64dd473ac41d None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:48.250994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 152/297] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:48.251462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ea2df-0a00-47fb-8808-cffebf659360 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:48.251865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ea2df-0a00-47fb-8808-cffebf659360 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:48.252023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ea2df-0a00-47fb-8808-cffebf659360 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:48.253453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 146/298] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:48.256470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f8e95-7860-43d1-85a7-b8a992e6fece None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:48.256994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f8e95-7860-43d1-85a7-b8a992e6fece None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:48.257468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90f8e95-7860-43d1-85a7-b8a992e6fece None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:48.265572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f972ef-d45a-419a-a1b9-f37fe535bda0 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:48.266318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f972ef-d45a-419a-a1b9-f37fe535bda0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:48.266318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f972ef-d45a-419a-a1b9-f37fe535bda0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:48.280938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b90f8e95-7860-43d1-85a7-b8a992e6fece None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:48.282456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b90f8e95-7860-43d1-85a7-b8a992e6fece None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:48.288030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-06f972ef-d45a-419a-a1b9-f37fe535bda0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:48.289207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-06f972ef-d45a-419a-a1b9-f37fe535bda0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:48.406776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 153/299] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:48.407121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 147/300] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:48.708437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90f8e95-7860-43d1-85a7-b8a992e6fece None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:48.759664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:48.843763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90f8e95-7860-43d1-85a7-b8a992e6fece None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:48.871434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f972ef-d45a-419a-a1b9-f37fe535bda0 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:48.898498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a92f404-c015-451b-9ac0-50c53ced3ec7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:48.899224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92f404-c015-451b-9ac0-50c53ced3ec7 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:48.899508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92f404-c015-451b-9ac0-50c53ced3ec7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:48.899785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a92f404-c015-451b-9ac0-50c53ced3ec7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:48.919378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a92f404-c015-451b-9ac0-50c53ced3ec7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-6a92f404-c015-451b-9ac0-50c53ced3ec7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:48.924907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:48.925762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:48.926970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:48.927604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 154/301] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:48.934474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a92f404-c015-451b-9ac0-50c53ced3ec7 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:49.048067 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f972ef-d45a-419a-a1b9-f37fe535bda0 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:49.083520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:49.091587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b4e5e0-127f-4ef9-9179-365412c34a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:49.091762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a92f404-c015-451b-9ac0-50c53ced3ec7 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:49.092632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4e5e0-127f-4ef9-9179-365412c34a81 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.093375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4e5e0-127f-4ef9-9179-365412c34a81 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.094899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:49.095547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b4e5e0-127f-4ef9-9179-365412c34a81 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.108107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b4e5e0-127f-4ef9-9179-365412c34a81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-e0b4e5e0-127f-4ef9-9179-365412c34a81 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:49.114600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:49.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:49.116304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 148/302] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:42:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:49.116938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412f4a2f-5a5b-4877-88fe-5dc3f831f01a None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.117184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412f4a2f-5a5b-4877-88fe-5dc3f831f01a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.117488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412f4a2f-5a5b-4877-88fe-5dc3f831f01a None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.123107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 149/303] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:49.123107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19520f0-e607-42c0-9748-ecf5cec425d6 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.123107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19520f0-e607-42c0-9748-ecf5cec425d6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.123107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19520f0-e607-42c0-9748-ecf5cec425d6 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.123107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 150/304] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:49 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:49.126993 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19520f0-e607-42c0-9748-ecf5cec425d6 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:49.140700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61de89d-7e03-4adc-8295-6fde63161041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:49.141375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61de89d-7e03-4adc-8295-6fde63161041 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.141647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61de89d-7e03-4adc-8295-6fde63161041 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.141920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61de89d-7e03-4adc-8295-6fde63161041 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.142590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61de89d-7e03-4adc-8295-6fde63161041 None admin] RBAC: Adding query filter params (name=member) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:49.142891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61de89d-7e03-4adc-8295-6fde63161041 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:49.144363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61de89d-7e03-4adc-8295-6fde63161041 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:49.155697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 155/305] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:49.157016 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94a8da-0110-4707-a593-277db817d192 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.157307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94a8da-0110-4707-a593-277db817d192 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.157577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a94a8da-0110-4707-a593-277db817d192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.180154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-8a94a8da-0110-4707-a593-277db817d192 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:49.181373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8a94a8da-0110-4707-a593-277db817d192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:49.245808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19520f0-e607-42c0-9748-ecf5cec425d6 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:49.260020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 156/306] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:49.261556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543dcdd5-6125-4263-bf9d-34719eab10fd None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.261950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543dcdd5-6125-4263-bf9d-34719eab10fd None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.262235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543dcdd5-6125-4263-bf9d-34719eab10fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.281496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-543dcdd5-6125-4263-bf9d-34719eab10fd None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:49.282688 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-543dcdd5-6125-4263-bf9d-34719eab10fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:49.288377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:49.289589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.290109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.290602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.292407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:49.292898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:49.294983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:49.306894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 151/307] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:42:49 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:49.307618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:49.368681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 157/308] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:49.370688 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543dcdd5-6125-4263-bf9d-34719eab10fd None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:49.378256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:49.407788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:49.451068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a4fa10-1505-4e13-b5ae-7771cf49f4f7 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:49.497163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543dcdd5-6125-4263-bf9d-34719eab10fd None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:49.497344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a081baa-2fff-44eb-a2f5-591f5f911377 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:49.497344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a081baa-2fff-44eb-a2f5-591f5f911377 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.497344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a081baa-2fff-44eb-a2f5-591f5f911377 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.497344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a081baa-2fff-44eb-a2f5-591f5f911377 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.539809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d9a388-2f85-48fa-b4f6-0776ecf5f23e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:49.540842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d9a388-2f85-48fa-b4f6-0776ecf5f23e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.541330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d9a388-2f85-48fa-b4f6-0776ecf5f23e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.541791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d9a388-2f85-48fa-b4f6-0776ecf5f23e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.582571 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a081baa-2fff-44eb-a2f5-591f5f911377 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:49.584272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a081baa-2fff-44eb-a2f5-591f5f911377 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:49.632196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d9a388-2f85-48fa-b4f6-0776ecf5f23e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:49.634038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d9a388-2f85-48fa-b4f6-0776ecf5f23e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:49.669400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:49.670360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:49.735046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 152/309] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:49.736238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a081baa-2fff-44eb-a2f5-591f5f911377 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:49.768530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 158/310] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:49.770400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d9a388-2f85-48fa-b4f6-0776ecf5f23e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:49.874198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a081baa-2fff-44eb-a2f5-591f5f911377 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:49.902641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d9a388-2f85-48fa-b4f6-0776ecf5f23e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:49.921871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25da2eb-1d98-420c-ba76-7386db6edf32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:49.922996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25da2eb-1d98-420c-ba76-7386db6edf32 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.923516 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25da2eb-1d98-420c-ba76-7386db6edf32 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.923878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25da2eb-1d98-420c-ba76-7386db6edf32 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.936732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25da2eb-1d98-420c-ba76-7386db6edf32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-e25da2eb-1d98-420c-ba76-7386db6edf32 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:49.941750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:49.942670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:49.943622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 153/311] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:42:49 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:49.944405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25da2eb-1d98-420c-ba76-7386db6edf32 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:49.945840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:49.957889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eaae10-b4a9-455b-bd07-37b6f67528a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:49.958738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eaae10-b4a9-455b-bd07-37b6f67528a2 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:49.959374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eaae10-b4a9-455b-bd07-37b6f67528a2 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:49.959652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38eaae10-b4a9-455b-bd07-37b6f67528a2 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:49.961082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eaae10-b4a9-455b-bd07-37b6f67528a2 None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:49.961762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:49.964503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38eaae10-b4a9-455b-bd07-37b6f67528a2 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:49.966703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 159/312] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:49 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:49.967901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eaae10-b4a9-455b-bd07-37b6f67528a2 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.079077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25da2eb-1d98-420c-ba76-7386db6edf32 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.101532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38eaae10-b4a9-455b-bd07-37b6f67528a2 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.127089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa2143-e382-46cf-aab1-b7357ea59490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.128290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa2143-e382-46cf-aab1-b7357ea59490 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.128290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa2143-e382-46cf-aab1-b7357ea59490 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.128290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fa2143-e382-46cf-aab1-b7357ea59490 None admin] PATH_INFO: `/v3/projects/ebd577f63e0b41edb25465e82d9af4df` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.130148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fa2143-e382-46cf-aab1-b7357ea59490 None admin] RBAC: Authorizing `identity:get_project(project_id=ebd577f63e0b41edb25465e82d9af4df)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:50.131719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fa2143-e382-46cf-aab1-b7357ea59490 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:50.133866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 154/313] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:49 2022] GET /identity/v3/projects/ebd577f63e0b41edb25465e82d9af4df => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:50.134912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa2143-e382-46cf-aab1-b7357ea59490 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.149165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3860dd02-9acf-4816-9639-ba94e91a6784 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.149807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3860dd02-9acf-4816-9639-ba94e91a6784 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.150091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3860dd02-9acf-4816-9639-ba94e91a6784 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.150684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3860dd02-9acf-4816-9639-ba94e91a6784 None admin] PATH_INFO: `/v3/roles/member` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.165138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3860dd02-9acf-4816-9639-ba94e91a6784 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-3860dd02-9acf-4816-9639-ba94e91a6784 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:50.170724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:50.171638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:50.172419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 160/314] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:50.173087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3860dd02-9acf-4816-9639-ba94e91a6784 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.250685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:50.252639 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:50.272432 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fa2143-e382-46cf-aab1-b7357ea59490 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.311446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3860dd02-9acf-4816-9639-ba94e91a6784 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.317569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.318186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.318465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.318733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.319754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:50.320272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:50.321935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:50.330906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 155/315] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:50.331765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.356305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f261dad-0300-4e80-a72c-41c11328125d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.356931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f261dad-0300-4e80-a72c-41c11328125d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.357207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f261dad-0300-4e80-a72c-41c11328125d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.357472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f261dad-0300-4e80-a72c-41c11328125d None admin] PATH_INFO: `/v3/groups/4959a820d03549248dc8bed9f38d3012` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.358590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f261dad-0300-4e80-a72c-41c11328125d None admin] RBAC: Authorizing `identity:get_group(group_id=4959a820d03549248dc8bed9f38d3012)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:50.360090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f261dad-0300-4e80-a72c-41c11328125d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:50.361834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 161/316] 10.222.0.20 () {56 vars in 1180 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/groups/4959a820d03549248dc8bed9f38d3012 => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:50.367291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f261dad-0300-4e80-a72c-41c11328125d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.445437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc0783c-888e-4f8b-8690-4024994926cf None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.482455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f261dad-0300-4e80-a72c-41c11328125d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.493287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3e7e3a-ec95-4e00-99f1-b03322992b9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.494356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e7e3a-ec95-4e00-99f1-b03322992b9b None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.494855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e7e3a-ec95-4e00-99f1-b03322992b9b None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.496001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3e7e3a-ec95-4e00-99f1-b03322992b9b None admin] PATH_INFO: `/v3/groups/4959a820d03549248dc8bed9f38d3012` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.497618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e7e3a-ec95-4e00-99f1-b03322992b9b None admin] RBAC: Authorizing `identity:get_group(group_id=4959a820d03549248dc8bed9f38d3012)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:50.500466 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c3e7e3a-ec95-4e00-99f1-b03322992b9b None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:50.501045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 156/317] 10.222.0.20 () {56 vars in 1180 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/groups/4959a820d03549248dc8bed9f38d3012 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:50.503415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d6836-607b-4956-be00-831c72b5b5a3 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.503415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d6836-607b-4956-be00-831c72b5b5a3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.503415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7d6836-607b-4956-be00-831c72b5b5a3 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.504657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 157/318] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:50.507733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7d6836-607b-4956-be00-831c72b5b5a3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.510717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:50.523198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:50.524555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.525212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.527495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.527495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.527495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] RBAC: Adding query filter params (name=member) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:50.527495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:50.528394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:50.537400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 162/319] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:50.539028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.627126 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7d6836-607b-4956-be00-831c72b5b5a3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.644579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e45185-102c-45eb-b8c1-b61c80f690b9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.659576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21bb1a9-8d5c-461b-be84-f2f9bc91009e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.660184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21bb1a9-8d5c-461b-be84-f2f9bc91009e None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.660465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21bb1a9-8d5c-461b-be84-f2f9bc91009e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.660737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21bb1a9-8d5c-461b-be84-f2f9bc91009e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.677765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.678358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.678634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.678907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.679655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] RBAC: Adding query filter params (group.id=4959a820d03549248dc8bed9f38d3012, role.id=974299ae4a414939a4f7e186921d4569, scope.project.id=ebd577f63e0b41edb25465e82d9af4df) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:50.679958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:50.681388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:50.687568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 163/320] 10.222.0.20 () {56 vars in 1376 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/role_assignments?group.id=4959a820d03549248dc8bed9f38d3012&scope.project.id=ebd577f63e0b41edb25465e82d9af4df&role.id=974299ae4a414939a4f7e186921d4569 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:50.688404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.742893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:50.744846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21bb1a9-8d5c-461b-be84-f2f9bc91009e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:50.746357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21bb1a9-8d5c-461b-be84-f2f9bc91009e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:50.759019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:50.806695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec328a1-94a9-4dd5-93d3-b2e679b5897f None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:50.845042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795da722-7e43-4aad-930a-77ffc88511ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:50.845685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795da722-7e43-4aad-930a-77ffc88511ea None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.845960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795da722-7e43-4aad-930a-77ffc88511ea None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.846228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795da722-7e43-4aad-930a-77ffc88511ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.847039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795da722-7e43-4aad-930a-77ffc88511ea None admin] RBAC: Adding query filter params (group.id=4959a820d03549248dc8bed9f38d3012, role.id=3cb31ad012324a57acd3a4bd258d1fb1, scope.project.id=ebd577f63e0b41edb25465e82d9af4df) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:50.847349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795da722-7e43-4aad-930a-77ffc88511ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:50.848944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795da722-7e43-4aad-930a-77ffc88511ea None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:50.863145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 164/321] 10.222.0.20 () {56 vars in 1376 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/role_assignments?group.id=4959a820d03549248dc8bed9f38d3012&scope.project.id=ebd577f63e0b41edb25465e82d9af4df&role.id=3cb31ad012324a57acd3a4bd258d1fb1 => generated 611 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:50.863145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125a6cf-be5d-45ca-89fa-edfbcc45c7b7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:50.863145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125a6cf-be5d-45ca-89fa-edfbcc45c7b7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:50.863145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125a6cf-be5d-45ca-89fa-edfbcc45c7b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:50.863608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 158/322] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:50.863608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21bb1a9-8d5c-461b-be84-f2f9bc91009e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.887609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5125a6cf-be5d-45ca-89fa-edfbcc45c7b7 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:50.891821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5125a6cf-be5d-45ca-89fa-edfbcc45c7b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:50.978006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 165/323] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:50.980122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5125a6cf-be5d-45ca-89fa-edfbcc45c7b7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:50.999635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21bb1a9-8d5c-461b-be84-f2f9bc91009e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.013654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:51.043662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056a32c5-0834-4d8a-832c-35a8903a3259 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.044278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056a32c5-0834-4d8a-832c-35a8903a3259 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.044577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056a32c5-0834-4d8a-832c-35a8903a3259 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.045794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056a32c5-0834-4d8a-832c-35a8903a3259 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.048898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:51.110457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5125a6cf-be5d-45ca-89fa-edfbcc45c7b7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.129362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056a32c5-0834-4d8a-832c-35a8903a3259 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.131037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056a32c5-0834-4d8a-832c-35a8903a3259 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.156324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a387795-0747-4e6e-8c3a-8d11b316c3a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.157012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a387795-0747-4e6e-8c3a-8d11b316c3a7 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.157294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a387795-0747-4e6e-8c3a-8d11b316c3a7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.157670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a387795-0747-4e6e-8c3a-8d11b316c3a7 None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.158799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a387795-0747-4e6e-8c3a-8d11b316c3a7 None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.160283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a387795-0747-4e6e-8c3a-8d11b316c3a7 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.162153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 166/324] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:51.167956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a387795-0747-4e6e-8c3a-8d11b316c3a7 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:51.258095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 159/325] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:51.264105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056a32c5-0834-4d8a-832c-35a8903a3259 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:51.277622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:51.292563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a387795-0747-4e6e-8c3a-8d11b316c3a7 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.305769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:51.335336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5cabd5-82c9-429c-a152-7a7f7061b2aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.336683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cabd5-82c9-429c-a152-7a7f7061b2aa None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.336952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cabd5-82c9-429c-a152-7a7f7061b2aa None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.337217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5cabd5-82c9-429c-a152-7a7f7061b2aa None admin] PATH_INFO: `/v3/groups/4959a820d03549248dc8bed9f38d3012` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.338366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5cabd5-82c9-429c-a152-7a7f7061b2aa None admin] RBAC: Authorizing `identity:get_group(group_id=4959a820d03549248dc8bed9f38d3012)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.340452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5cabd5-82c9-429c-a152-7a7f7061b2aa None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.343418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 167/326] 10.222.0.20 () {56 vars in 1180 bytes} [Wed Oct 19 21:42:51 2022] GET /identity/v3/groups/4959a820d03549248dc8bed9f38d3012 => generated 226 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:51.343418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5cabd5-82c9-429c-a152-7a7f7061b2aa None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:51.390193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056a32c5-0834-4d8a-832c-35a8903a3259 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.428895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d994dcc4-772e-448e-990e-7dd5bd2c3815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.429758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d994dcc4-772e-448e-990e-7dd5bd2c3815 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.430184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d994dcc4-772e-448e-990e-7dd5bd2c3815 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.430637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d994dcc4-772e-448e-990e-7dd5bd2c3815 None admin] PATH_INFO: `/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.431908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d994dcc4-772e-448e-990e-7dd5bd2c3815 None admin] RBAC: Authorizing `identity:get_project(project_id=1c9d2e1ab4424a98b5c3cd424a58ec18)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.433943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d994dcc4-772e-448e-990e-7dd5bd2c3815 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.436005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 160/327] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:51 2022] GET /identity/v3/projects/1c9d2e1ab4424a98b5c3cd424a58ec18 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:51.436805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d994dcc4-772e-448e-990e-7dd5bd2c3815 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:51.464231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5cabd5-82c9-429c-a152-7a7f7061b2aa None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.499571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd28f93d-a27c-4cb4-af14-8c3dad9098f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.500205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd28f93d-a27c-4cb4-af14-8c3dad9098f9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.500480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd28f93d-a27c-4cb4-af14-8c3dad9098f9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.500753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd28f93d-a27c-4cb4-af14-8c3dad9098f9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.510524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd28f93d-a27c-4cb4-af14-8c3dad9098f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-bd28f93d-a27c-4cb4-af14-8c3dad9098f9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:51.515811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:51.516813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:51.517591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 168/328] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:42:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:51.518231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd28f93d-a27c-4cb4-af14-8c3dad9098f9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:51.522177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:51.543510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d994dcc4-772e-448e-990e-7dd5bd2c3815 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.557160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:51.575941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.576556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.577205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.577655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.578809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] RBAC: Adding query filter params (group.id=4959a820d03549248dc8bed9f38d3012, role.id=3cb31ad012324a57acd3a4bd258d1fb1, scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:51.579159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.580612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.586440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 161/329] 10.222.0.20 () {56 vars in 1376 bytes} [Wed Oct 19 21:42:51 2022] GET /identity/v3/role_assignments?group.id=4959a820d03549248dc8bed9f38d3012&scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18&role.id=3cb31ad012324a57acd3a4bd258d1fb1 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:51.587255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:51.624435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd28f93d-a27c-4cb4-af14-8c3dad9098f9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.664117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa84b38-31b3-4e79-bc15-1a64e5527f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.664884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa84b38-31b3-4e79-bc15-1a64e5527f6a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.665162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa84b38-31b3-4e79-bc15-1a64e5527f6a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.665429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa84b38-31b3-4e79-bc15-1a64e5527f6a None admin] PATH_INFO: `/v3/groups/4959a820d03549248dc8bed9f38d3012` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.666602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa84b38-31b3-4e79-bc15-1a64e5527f6a None admin] RBAC: Authorizing `identity:get_group(group_id=4959a820d03549248dc8bed9f38d3012)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.668730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fa84b38-31b3-4e79-bc15-1a64e5527f6a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.670375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 169/330] 10.222.0.20 () {56 vars in 1180 bytes} [Wed Oct 19 21:42:51 2022] GET /identity/v3/groups/4959a820d03549248dc8bed9f38d3012 => generated 226 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:51.676345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa84b38-31b3-4e79-bc15-1a64e5527f6a None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:51.704882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c5d4be-80db-47bc-bf3e-b5be82328785 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.746036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.746659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.746939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.747240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.747922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:51.748237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.749647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.758754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 162/331] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:42:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:51.764973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:51.770042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:51.788669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fa84b38-31b3-4e79-bc15-1a64e5527f6a None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.800196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:51.829998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.830597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.830869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.831229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.831952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] RBAC: Adding query filter params (group.id=4959a820d03549248dc8bed9f38d3012, role.id=974299ae4a414939a4f7e186921d4569, scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:51.832252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.833687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.839515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 170/332] 10.222.0.20 () {56 vars in 1376 bytes} [Wed Oct 19 21:42:51 2022] GET /identity/v3/role_assignments?group.id=4959a820d03549248dc8bed9f38d3012&scope.project.id=1c9d2e1ab4424a98b5c3cd424a58ec18&role.id=974299ae4a414939a4f7e186921d4569 => generated 611 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:51.882860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd403b3e-7de7-4488-8e52-93de17c3a695 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:51.920241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8802591d-23da-475b-a598-3cb44166e28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:51.920974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8802591d-23da-475b-a598-3cb44166e28e None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:51.921281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8802591d-23da-475b-a598-3cb44166e28e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:51.921553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8802591d-23da-475b-a598-3cb44166e28e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:51.991551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8802591d-23da-475b-a598-3cb44166e28e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:51.993027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8802591d-23da-475b-a598-3cb44166e28e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:51.996761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:52.095703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 163/333] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:42:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:42:52.101420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:52.202532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6380fcae-7b7a-4e48-9462-430a62d64871 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:52.235727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337634df-fcdf-47d1-b614-b77e7ae647c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:52.236368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337634df-fcdf-47d1-b614-b77e7ae647c4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:52.236640 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337634df-fcdf-47d1-b614-b77e7ae647c4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:52.236916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337634df-fcdf-47d1-b614-b77e7ae647c4 None admin] PATH_INFO: `/v3/projects/0fd8fb8e58c74c8cbe244194ce71b304` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:52.238033 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337634df-fcdf-47d1-b614-b77e7ae647c4 None admin] RBAC: Authorizing `identity:get_project(project_id=0fd8fb8e58c74c8cbe244194ce71b304)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:52.239511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337634df-fcdf-47d1-b614-b77e7ae647c4 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:52.241137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 171/334] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:42:52 2022] GET /identity/v3/projects/0fd8fb8e58c74c8cbe244194ce71b304 => generated 344 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:52.246841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8802591d-23da-475b-a598-3cb44166e28e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:52.349484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8802591d-23da-475b-a598-3cb44166e28e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:52.351720 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:52.382665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3189e5-b8b8-4880-bd8d-e0f241200783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:52.383323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3189e5-b8b8-4880-bd8d-e0f241200783 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:52.383803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3189e5-b8b8-4880-bd8d-e0f241200783 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:52.384096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3189e5-b8b8-4880-bd8d-e0f241200783 None admin] PATH_INFO: `/v3/groups/6e52655036924ac5a48cde7936bc03c8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:52.385880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3189e5-b8b8-4880-bd8d-e0f241200783 None admin] RBAC: Authorizing `identity:get_group(group_id=6e52655036924ac5a48cde7936bc03c8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:52.387341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3189e5-b8b8-4880-bd8d-e0f241200783 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:52.388907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 164/335] 10.222.0.20 () {56 vars in 1180 bytes} [Wed Oct 19 21:42:52 2022] GET /identity/v3/groups/6e52655036924ac5a48cde7936bc03c8 => generated 229 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:52.394701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337634df-fcdf-47d1-b614-b77e7ae647c4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:52.433600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:52.489930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337634df-fcdf-47d1-b614-b77e7ae647c4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:52.521431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e6ca9-f12d-4ed2-a3b3-3c7e7b684470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:52.522013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e6ca9-f12d-4ed2-a3b3-3c7e7b684470 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:52.522286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e6ca9-f12d-4ed2-a3b3-3c7e7b684470 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:52.522553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e6ca9-f12d-4ed2-a3b3-3c7e7b684470 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:52.523290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7e6ca9-f12d-4ed2-a3b3-3c7e7b684470 None admin] RBAC: Adding query filter params (group.id=6e52655036924ac5a48cde7936bc03c8, role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=0fd8fb8e58c74c8cbe244194ce71b304) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:52.523592 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7e6ca9-f12d-4ed2-a3b3-3c7e7b684470 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:52.525005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f7e6ca9-f12d-4ed2-a3b3-3c7e7b684470 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:52.531268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 172/336] 10.222.0.20 () {56 vars in 1376 bytes} [Wed Oct 19 21:42:52 2022] GET /identity/v3/role_assignments?group.id=6e52655036924ac5a48cde7936bc03c8&scope.project.id=0fd8fb8e58c74c8cbe244194ce71b304&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 611 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:55.373465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e670782-aa13-4485-9c6d-c400c7cb741d None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:55.373986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e670782-aa13-4485-9c6d-c400c7cb741d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:55.373986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e670782-aa13-4485-9c6d-c400c7cb741d None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:55.375668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 165/337] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:55.389905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53806386-1946-46a1-bf89-a75d6989ddde None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:55.391482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53806386-1946-46a1-bf89-a75d6989ddde None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:55.391755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53806386-1946-46a1-bf89-a75d6989ddde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:55.406735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7a7a7-396b-4263-98bd-2ac633e3f836 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:55.407049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7a7a7-396b-4263-98bd-2ac633e3f836 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:55.407301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf7a7a7-396b-4263-98bd-2ac633e3f836 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:55.408760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 166/338] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:42:55.413623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7732b8-5c21-4069-8066-714ce64b9965 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:55.413896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7732b8-5c21-4069-8066-714ce64b9965 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:55.414161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7732b8-5c21-4069-8066-714ce64b9965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:55.433161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-53806386-1946-46a1-bf89-a75d6989ddde None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:55.434350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-53806386-1946-46a1-bf89-a75d6989ddde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:55.444590 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-dd7732b8-5c21-4069-8066-714ce64b9965 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:55.445758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dd7732b8-5c21-4069-8066-714ce64b9965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:55.547414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 173/339] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:55.579015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 167/340] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:42:55.664535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a569f47-30ce-4dbd-8874-0332a1bda6bc None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:55.664808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a569f47-30ce-4dbd-8874-0332a1bda6bc None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:55.665001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a569f47-30ce-4dbd-8874-0332a1bda6bc None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:55.666590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 174/341] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:55.672292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f32a81f-7b9f-4eb3-bc21-a9a4749b10e5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:55.672582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f32a81f-7b9f-4eb3-bc21-a9a4749b10e5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:55.672861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f32a81f-7b9f-4eb3-bc21-a9a4749b10e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:55.699764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3f32a81f-7b9f-4eb3-bc21-a9a4749b10e5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:55.700971 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3f32a81f-7b9f-4eb3-bc21-a9a4749b10e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:55.780885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 168/342] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:55.870855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a569f47-30ce-4dbd-8874-0332a1bda6bc None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:55.881307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f32a81f-7b9f-4eb3-bc21-a9a4749b10e5 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:55.916089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:55.939320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:55.992321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a569f47-30ce-4dbd-8874-0332a1bda6bc None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.030776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.031844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.032266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.032683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.046169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.047960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:42:56.066088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:56.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:42:56.068310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:42:56.068310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:56.068310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:56.070052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 175/343] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:56.077007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:56.081737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f32a81f-7b9f-4eb3-bc21-a9a4749b10e5 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.147578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.148218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.148506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.148799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.160222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.161824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:42:56.177825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:56.178866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:42:56.179763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:42:56.179763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:56.179763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:56.179763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 169/344] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:56.179763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:56.182165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:56.201051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3477703d-8ca6-4c0f-ad55-89175e83ccfd None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.236285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eacdebb-88a3-4dd3-b4de-4ca73a820f23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.237164 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacdebb-88a3-4dd3-b4de-4ca73a820f23 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.237620 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacdebb-88a3-4dd3-b4de-4ca73a820f23 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.237887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eacdebb-88a3-4dd3-b4de-4ca73a820f23 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.238531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eacdebb-88a3-4dd3-b4de-4ca73a820f23 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:56.238831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eacdebb-88a3-4dd3-b4de-4ca73a820f23 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.240271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eacdebb-88a3-4dd3-b4de-4ca73a820f23 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.253533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 176/345] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:42:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:42:56.253533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b68b97-01a0-4e5a-a0d6-aff6445695fe None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.253533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b68b97-01a0-4e5a-a0d6-aff6445695fe None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.253533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b68b97-01a0-4e5a-a0d6-aff6445695fe None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.253533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 177/346] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:56 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:56.262010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b68b97-01a0-4e5a-a0d6-aff6445695fe None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:56.273378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:56.288681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5193e28a-2499-4c07-96ba-e7de2a7a5d4f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.326072 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.326808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.327135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.327481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.338060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.340165 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:42:56.359214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:56.360290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:42:56.361303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:42:56.361303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:56.361303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:56.362393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 170/347] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:56.363246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:56.373930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b68b97-01a0-4e5a-a0d6-aff6445695fe None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.410696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f76a2a47-2403-4d02-9fe4-dd6fe431d70e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.411394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a2a47-2403-4d02-9fe4-dd6fe431d70e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.411669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a2a47-2403-4d02-9fe4-dd6fe431d70e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.411999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f76a2a47-2403-4d02-9fe4-dd6fe431d70e None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.412639 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76a2a47-2403-4d02-9fe4-dd6fe431d70e None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:56.412939 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76a2a47-2403-4d02-9fe4-dd6fe431d70e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.414455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f76a2a47-2403-4d02-9fe4-dd6fe431d70e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.426056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 178/348] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:42:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:56.427251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e897b265-aeea-44ab-b14f-0746d017ad5f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.427533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e897b265-aeea-44ab-b14f-0746d017ad5f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.427802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e897b265-aeea-44ab-b14f-0746d017ad5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.436170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:56.447205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e897b265-aeea-44ab-b14f-0746d017ad5f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:56.448409 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e897b265-aeea-44ab-b14f-0746d017ad5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:56.471983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b39c0f-2fce-41cd-b6d7-cf242d064b28 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.512992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0388e7bb-ce98-451d-b930-c1509451bad3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.513928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0388e7bb-ce98-451d-b930-c1509451bad3 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.514606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0388e7bb-ce98-451d-b930-c1509451bad3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.515110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0388e7bb-ce98-451d-b930-c1509451bad3 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.516315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0388e7bb-ce98-451d-b930-c1509451bad3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.518196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0388e7bb-ce98-451d-b930-c1509451bad3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.520931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 179/349] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:56.521600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e897b265-aeea-44ab-b14f-0746d017ad5f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:56.542120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:56.604416 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 171/350] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:56 2022] POST /identity/v3/users => generated 246 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:56.606465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0388e7bb-ce98-451d-b930-c1509451bad3 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:56.643930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e897b265-aeea-44ab-b14f-0746d017ad5f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.708921 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.709554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.709835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.710107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.712516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:56.712830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.716596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.723853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:56.744466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 180/351] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:42:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:56.750478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:56.801496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0388e7bb-ce98-451d-b930-c1509451bad3 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.841241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c61479c-b359-4143-a906-49d23328fde6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.841241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61479c-b359-4143-a906-49d23328fde6 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.841501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61479c-b359-4143-a906-49d23328fde6 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.841568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61479c-b359-4143-a906-49d23328fde6 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.842223 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c61479c-b359-4143-a906-49d23328fde6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.843697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c61479c-b359-4143-a906-49d23328fde6 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.868634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b419e0-57ed-464f-a2f2-94c8c15017fb None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:56.903627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdec4dc-7a7e-4e10-8764-38f666c006a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:56.915112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 172/352] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:56 2022] POST /identity/v3/users => generated 248 bytes in 299 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:56.915309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdec4dc-7a7e-4e10-8764-38f666c006a8 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.915309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdec4dc-7a7e-4e10-8764-38f666c006a8 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.915309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdec4dc-7a7e-4e10-8764-38f666c006a8 None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.915309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bdec4dc-7a7e-4e10-8764-38f666c006a8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:56.915623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7764048-cdbd-497e-838f-501d834839f8 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:56.915623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7764048-cdbd-497e-838f-501d834839f8 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:56.915623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7764048-cdbd-497e-838f-501d834839f8 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:56.915839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bdec4dc-7a7e-4e10-8764-38f666c006a8 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:56.915925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 173/353] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:56 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:56.915925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7764048-cdbd-497e-838f-501d834839f8 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:56.976359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:56.999016 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 181/354] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:56 2022] POST /identity/v3/users => generated 248 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:42:57.003172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a254dd-f3de-43b5-acdb-c60bd64d99eb None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:57.003482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a254dd-f3de-43b5-acdb-c60bd64d99eb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:57.003715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a254dd-f3de-43b5-acdb-c60bd64d99eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:57.041771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-40a254dd-f3de-43b5-acdb-c60bd64d99eb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:57.044348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-40a254dd-f3de-43b5-acdb-c60bd64d99eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:57.087058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7764048-cdbd-497e-838f-501d834839f8 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:57.129923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:57.136538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:57.137084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:57.137444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:57.139344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:57.153278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:57.155826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:57.160115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 182/355] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:42:57.172519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:57.174559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:42:57.175430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:42:57.175430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:42:57.175430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:57.176376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 174/356] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:42:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:57.183257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a254dd-f3de-43b5-acdb-c60bd64d99eb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:57.292529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a254dd-f3de-43b5-acdb-c60bd64d99eb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:57.329227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:57.329838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:57.330118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:57.330388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:57.331059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:57.331359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:57.332787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:57.346084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 183/357] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:42:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:57.352063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:57.449825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:57.463080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:57.467962 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a054b8-ec36-4c54-afed-b82bf9d140f0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:57.512813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8166e8-3800-406d-b18c-b9bfd914c75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:57.513436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8166e8-3800-406d-b18c-b9bfd914c75a None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:57.513677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8166e8-3800-406d-b18c-b9bfd914c75a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:57.513946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8166e8-3800-406d-b18c-b9bfd914c75a None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:57.514603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8166e8-3800-406d-b18c-b9bfd914c75a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:57.519360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8166e8-3800-406d-b18c-b9bfd914c75a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:57.541413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:57.608998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4013884-c414-4467-887e-77260ae766c3 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:57.653412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 175/358] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:42:57 2022] POST /identity/v3/users => generated 249 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:42:57.658194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:57.658900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:57.659209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:57.659479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:57.660199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] RBAC: Authorizing `identity:create_role()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:57.661795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:57.708703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.sql.core [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 21:42:57.708703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 19 21:42:57.708703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [parameters: {'id': 'f4ae137b5ba54e1d8ddad7477bfc34a7', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 19 21:42:57.708703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=83021) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result.read() Oct 19 21:42:57.716901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return ref.to_dict() Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:42:57.718890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application yield resource Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise self.value Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application session.commit() Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 21:42:57.721122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.session.flush() Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application compat.raise_( Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 21:42:57.723222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:42:57.725279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result.read() Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:42:57.727212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [parameters: {'id': 'f4ae137b5ba54e1d8ddad7477bfc34a7', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:57.728859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 19 21:42:57.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:42:57.739524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 184/359] 10.222.0.20 () {60 vars in 1131 bytes} [Wed Oct 19 21:42:57 2022] POST /identity/v3/roles => generated 144 bytes in 269 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:42:57.739673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8166e8-3800-406d-b18c-b9bfd914c75a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:57.837674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8166e8-3800-406d-b18c-b9bfd914c75a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:57.875161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dee225e-fe26-4280-8396-49ee755dbde3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:57.876095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dee225e-fe26-4280-8396-49ee755dbde3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:57.876545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dee225e-fe26-4280-8396-49ee755dbde3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:57.876984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dee225e-fe26-4280-8396-49ee755dbde3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:57.888522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dee225e-fe26-4280-8396-49ee755dbde3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-8dee225e-fe26-4280-8396-49ee755dbde3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:57.895253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:57.896201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 19 21:42:57.897098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:57.899150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 176/360] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:42:57 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:57.908361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:58.020457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb73754c-96de-4c18-b2ce-ca7de2d578ab None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:58.023060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:58.063623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447f27b7-c07e-4dcf-95cf-0355c6af120e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:58.064624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447f27b7-c07e-4dcf-95cf-0355c6af120e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:58.065159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447f27b7-c07e-4dcf-95cf-0355c6af120e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:58.065570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447f27b7-c07e-4dcf-95cf-0355c6af120e None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:58.066411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447f27b7-c07e-4dcf-95cf-0355c6af120e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:58.066898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447f27b7-c07e-4dcf-95cf-0355c6af120e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:58.070405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447f27b7-c07e-4dcf-95cf-0355c6af120e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:58.082006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 185/361] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:42:57 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:59.079410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d340d9d-eb84-4ef5-aaaa-31c63d633cdc None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:59.080398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d340d9d-eb84-4ef5-aaaa-31c63d633cdc None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:59.080879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d340d9d-eb84-4ef5-aaaa-31c63d633cdc None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:59.082674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 177/362] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:59.087834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de484354-4d92-48a8-849c-af6e4fa9ea86 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:59.088664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de484354-4d92-48a8-849c-af6e4fa9ea86 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:59.089798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de484354-4d92-48a8-849c-af6e4fa9ea86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:59.115057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-de484354-4d92-48a8-849c-af6e4fa9ea86 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:59.116619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-de484354-4d92-48a8-849c-af6e4fa9ea86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:59.197463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 186/363] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:59.459915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3927c8-7b25-4d2c-a28c-860c7fc911d3 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:59.460524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3927c8-7b25-4d2c-a28c-860c7fc911d3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:59.461012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3927c8-7b25-4d2c-a28c-860c7fc911d3 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:59.462526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 178/364] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:59.468420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc3d1b3-2348-4bef-9f0b-3dd86fa925ba None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:59.468922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc3d1b3-2348-4bef-9f0b-3dd86fa925ba None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:59.469407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc3d1b3-2348-4bef-9f0b-3dd86fa925ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:59.494135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ebc3d1b3-2348-4bef-9f0b-3dd86fa925ba None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:59.495800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ebc3d1b3-2348-4bef-9f0b-3dd86fa925ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:42:59.519237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3927c8-7b25-4d2c-a28c-860c7fc911d3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:59.551306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:59.587406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 187/365] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:42:59.659298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3927c8-7b25-4d2c-a28c-860c7fc911d3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:59.697487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d388469-a1c2-4dfd-950d-67988e572592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:59.698177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d388469-a1c2-4dfd-950d-67988e572592 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:59.698979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d388469-a1c2-4dfd-950d-67988e572592 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:59.699263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d388469-a1c2-4dfd-950d-67988e572592 None admin] PATH_INFO: `/v3/roles/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:59.711175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d388469-a1c2-4dfd-950d-67988e572592 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-1d388469-a1c2-4dfd-950d-67988e572592 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:42:59.716402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:42:59.717274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:42:59.718039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 179/366] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:42:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:42:59.723447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc3d1b3-2348-4bef-9f0b-3dd86fa925ba None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:59.759910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:42:59.863483 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc3d1b3-2348-4bef-9f0b-3dd86fa925ba None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:42:59.905296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:42:59.905896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:59.906175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:59.906446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:59.907151 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:42:59.907457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:42:59.908828 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:42:59.915426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cd9c62-ae9f-4dd0-a633-39b8a5128d09 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:59.915709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cd9c62-ae9f-4dd0-a633-39b8a5128d09 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:59.915984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cd9c62-ae9f-4dd0-a633-39b8a5128d09 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:59.917371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 180/367] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:42:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:42:59.923874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d26e3d-7c22-4bda-972f-dc437579e55a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:42:59.924213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d26e3d-7c22-4bda-972f-dc437579e55a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:42:59.924487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d26e3d-7c22-4bda-972f-dc437579e55a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:42:59.927790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 188/368] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:42:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:42:59.929452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:42:59.946351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-03d26e3d-7c22-4bda-972f-dc437579e55a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:42:59.947704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-03d26e3d-7c22-4bda-972f-dc437579e55a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:00.032451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:00.044405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ff56ab-16e2-4fd7-b63d-0d1d851aaca3 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.055006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 181/369] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:42:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:00.056096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d26e3d-7c22-4bda-972f-dc437579e55a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.068226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:00.111461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be480c4-26b8-4ef8-a66a-19be3c081ee9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.112212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be480c4-26b8-4ef8-a66a-19be3c081ee9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.112451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be480c4-26b8-4ef8-a66a-19be3c081ee9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.112730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be480c4-26b8-4ef8-a66a-19be3c081ee9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.135424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be480c4-26b8-4ef8-a66a-19be3c081ee9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-7be480c4-26b8-4ef8-a66a-19be3c081ee9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:00.141391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:00.142311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:00.143183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 189/370] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:42:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:00.148554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be480c4-26b8-4ef8-a66a-19be3c081ee9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.249746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d26e3d-7c22-4bda-972f-dc437579e55a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.304502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be480c4-26b8-4ef8-a66a-19be3c081ee9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.314864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.315548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.315829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.316105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.333236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:00.335160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:00.354583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:00.355549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:00.356371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:00.356371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:00.356371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:00.356371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 182/371] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:00.363858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.365144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.366046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.366533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.367050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.368157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:00.368673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:00.371970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:00.383212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 190/372] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:00.384496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.406743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:00.449593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:00.506895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5f4f34-0393-4f46-b5dc-48bd57394115 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.510233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639e1e77-fcdb-43cb-8a8a-4fdb37fd9871 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.549143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eec039-d8be-43fd-83e2-9cc99e59d814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.549799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eec039-d8be-43fd-83e2-9cc99e59d814 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.550076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eec039-d8be-43fd-83e2-9cc99e59d814 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.550344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26eec039-d8be-43fd-83e2-9cc99e59d814 None admin] PATH_INFO: `/v3/roles/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.563656 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26eec039-d8be-43fd-83e2-9cc99e59d814 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:00.566415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.567417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.567902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.568420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-26eec039-d8be-43fd-83e2-9cc99e59d814 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:00.569209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:00.570060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:00.570862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 191/373] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:00.572165 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:00.572700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:00.575064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:00.575257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eec039-d8be-43fd-83e2-9cc99e59d814 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.592517 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 183/374] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:00.593216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.654907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:00.687965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26eec039-d8be-43fd-83e2-9cc99e59d814 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.713718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d7bba1-e77b-4468-b2ce-17990be8e12f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.723712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.724333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.724619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.724896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.732712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:00.736026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:00.737479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:00.749769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b1950-6ed7-4abc-99e8-e6bb9a231b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.750371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b1950-6ed7-4abc-99e8-e6bb9a231b62 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.750646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b1950-6ed7-4abc-99e8-e6bb9a231b62 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.750980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b1950-6ed7-4abc-99e8-e6bb9a231b62 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.751802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b1950-6ed7-4abc-99e8-e6bb9a231b62 None admin] RBAC: Adding query filter params (name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:00.752105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b1950-6ed7-4abc-99e8-e6bb9a231b62 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:00.752589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:00.753765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:00.755049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:00.755049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:00.755049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:00.755240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b1950-6ed7-4abc-99e8-e6bb9a231b62 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:00.755315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 192/375] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:00.759000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.770789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 184/376] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:00.770789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a8f837-5a49-4495-8186-38e6a44b583b None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.770789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a8f837-5a49-4495-8186-38e6a44b583b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.770789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a8f837-5a49-4495-8186-38e6a44b583b None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.770789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 185/377] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:00.770789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a8f837-5a49-4495-8186-38e6a44b583b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.872313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc6736d-e44b-4f00-8af9-d65fb65f97b6 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.890129 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a8f837-5a49-4495-8186-38e6a44b583b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:00.895822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:00.908069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61228b2f-b7ab-48be-8508-39e6f9650a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.909181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61228b2f-b7ab-48be-8508-39e6f9650a6e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.909181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61228b2f-b7ab-48be-8508-39e6f9650a6e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.909356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61228b2f-b7ab-48be-8508-39e6f9650a6e None admin] PATH_INFO: `/v3/users/nova` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.916949 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61228b2f-b7ab-48be-8508-39e6f9650a6e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:00.918447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61228b2f-b7ab-48be-8508-39e6f9650a6e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:00.926046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:00.926652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.926925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.927235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.927880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:00.928335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:00.929704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-61228b2f-b7ab-48be-8508-39e6f9650a6e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:00.930964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:00.931818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:00.932742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 193/378] 10.222.0.20 () {56 vars in 1093 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/users/nova => generated 82 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:00.933334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93c5234-33dc-4a9c-9e21-3f42f2f57d05 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:00.933875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93c5234-33dc-4a9c-9e21-3f42f2f57d05 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:00.933875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93c5234-33dc-4a9c-9e21-3f42f2f57d05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:00.947342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 186/379] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:00.948754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:00.954015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a93c5234-33dc-4a9c-9e21-3f42f2f57d05 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:00.955255 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a93c5234-33dc-4a9c-9e21-3f42f2f57d05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:00.996992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.021165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 194/380] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:01.022122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93c5234-33dc-4a9c-9e21-3f42f2f57d05 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.055128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a77aee-5850-416e-b5a3-52c965c9bb45 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.090285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dabe87-8eef-4df9-acb8-2a9216ee3ee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.090906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dabe87-8eef-4df9-acb8-2a9216ee3ee5 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.091314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dabe87-8eef-4df9-acb8-2a9216ee3ee5 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.091581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dabe87-8eef-4df9-acb8-2a9216ee3ee5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.102412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dabe87-8eef-4df9-acb8-2a9216ee3ee5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.103989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dabe87-8eef-4df9-acb8-2a9216ee3ee5 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c9dabe87-8eef-4df9-acb8-2a9216ee3ee5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:01.121876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:01.122822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:01.123730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:01.123730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:01.123730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:01.123892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93c5234-33dc-4a9c-9e21-3f42f2f57d05 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.127713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 187/381] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:01.128810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc2dfd-dc80-44ec-abba-e837e294a68c None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.129103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc2dfd-dc80-44ec-abba-e837e294a68c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.129382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacc2dfd-dc80-44ec-abba-e837e294a68c None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.130764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 188/382] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:01.131422 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacc2dfd-dc80-44ec-abba-e837e294a68c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.143320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.166698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.167901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.168182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.168456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.169184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:01.169488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.171097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.215902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 195/383] 10.222.0.20 () {56 vars in 1133 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:01.215902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.225942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.237312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aacc2dfd-dc80-44ec-abba-e837e294a68c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.271024 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6e15f6-ecdd-4397-ae5e-5e14306de7e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.271676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e15f6-ecdd-4397-ae5e-5e14306de7e5 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.271920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e15f6-ecdd-4397-ae5e-5e14306de7e5 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.272196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6e15f6-ecdd-4397-ae5e-5e14306de7e5 None admin] PATH_INFO: `/v3/users/glance` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.281211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6e15f6-ecdd-4397-ae5e-5e14306de7e5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.282670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6e15f6-ecdd-4397-ae5e-5e14306de7e5 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-da6e15f6-ecdd-4397-ae5e-5e14306de7e5 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:01.295925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:01.297026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 189/384] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/users/glance => generated 84 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975445ba-def2-4d27-aaf6-838bfe9071d6 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.302544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975445ba-def2-4d27-aaf6-838bfe9071d6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.303406 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975445ba-def2-4d27-aaf6-838bfe9071d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.316180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-975445ba-def2-4d27-aaf6-838bfe9071d6 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:01.317369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-975445ba-def2-4d27-aaf6-838bfe9071d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:01.329095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b302878c-a70e-4a3f-a5ce-c6962e6afd93 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.363148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.363742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.364019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.364289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.364928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:01.365220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.366646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.378085 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 196/385] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:01.378808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.384025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 190/386] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:01.389783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975445ba-def2-4d27-aaf6-838bfe9071d6 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.419888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.468172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.487981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975445ba-def2-4d27-aaf6-838bfe9071d6 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.494563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2dd947-1b2e-43f6-8e43-2a7c4fd15a40 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.524253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da83f8b9-efb1-4f25-be2c-a7a3f127851a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.524879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83f8b9-efb1-4f25-be2c-a7a3f127851a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.525161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83f8b9-efb1-4f25-be2c-a7a3f127851a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.525442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da83f8b9-efb1-4f25-be2c-a7a3f127851a None admin] PATH_INFO: `/v3/roles/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.531533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.532153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.532429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.532699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.536589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da83f8b9-efb1-4f25-be2c-a7a3f127851a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-da83f8b9-efb1-4f25-be2c-a7a3f127851a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:01.541532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:01.542440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:01.543268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 191/387] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:01.543927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da83f8b9-efb1-4f25-be2c-a7a3f127851a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.549041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.550523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:01.564629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:01.565550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:01.566414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:01.566414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:01.566414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:01.566414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 197/388] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:01.567093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.650652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.659596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da83f8b9-efb1-4f25-be2c-a7a3f127851a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.669267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009b93a1-73e1-4afa-839b-80d90bad5fd5 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.692341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.693023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.693371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.693672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.694411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:01.694718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.696215 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.702801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.703451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.703734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.704009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.711312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.712753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:01.724148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:01.725018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:01.725881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 198/389] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/users/cinder => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:01.726457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.729187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.733542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 192/390] 10.222.0.20 () {56 vars in 1137 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:01.734345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.825824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975079fe-372b-44bc-9c99-0e22b72a87c0 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.834932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f398c86-f298-499d-af5f-c003e7054107 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.858315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.858923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.859230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.859509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.860176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:01.860480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.861850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.866743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:01.867374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:01.867666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:01.867918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:01.868550 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:01.868847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:01.869622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 199/391] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:01.871085 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.874315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.875239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:01.891881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 193/392] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:01.891881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:01.945046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:01.969323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d24ddff2-c9be-4f81-962f-2b1804818e04 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:01.984671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fd32d2-f07b-4434-8868-3d74f8eb27d0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.001630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.002281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.002564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.002845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.013890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.015386 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.018351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.018946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.019249 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.019520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.020265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:02.020568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.021973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:02.030814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:02.031706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:02.032644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:02.032644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:02.032644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:02.032644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 200/393] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:02.032863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.044927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 194/394] 10.222.0.20 () {56 vars in 1137 bytes} [Wed Oct 19 21:43:01 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:02.045745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.091375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:02.133607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdba0dc-2d39-4a5f-8432-cec39ba83118 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.140255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d46156-9a23-4b3a-ade4-9b98258b4fc6 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.158630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:02.167265 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.167890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.168144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.168415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.175309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.175925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.176241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.176755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.178809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.180300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.186692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.188171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:02.193792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:02.194731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:02.196154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:02.196154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:02.196154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:02.196154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 201/395] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:02.196154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:02.205523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:02.206565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:02.207641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:02.207641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:02.207641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:02.207641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 195/396] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:02.208376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.298069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da1e362-7764-4402-8a63-554504f66bae None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.306157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8ed520-4c83-4cf2-afa9-6019c0d8a1de None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.306997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:02.331840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.332460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.332736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.333007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.339418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.339552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.340010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.340306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.340574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.341003 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.341208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:02.341505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.342860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:02.352394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:02.353402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:02.354888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 202/397] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:02.355559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 196/398] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:02.356297 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.359341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.388004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:02.456053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace0ba45-e7b4-48a9-9b7f-26dcb619f5a6 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.456648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d962df-3bb4-441b-b1f0-530f9607668f None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.488986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.489427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.489634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.489916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.490018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.490190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.490277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.490547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.491229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:02.491529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.492915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.500447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.501887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.503821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 197/399] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:02.504600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:02.516952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:02.517917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:02.518741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:02.518741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:02.518741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:02.518741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 203/400] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:02.518987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.540928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:02.600647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:02.609961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413513bd-cc67-4bf9-9ff6-8e7481e8ed0f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.617716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2866ccf9-007a-427a-96eb-4fb4e2890514 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.643951 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.644552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.644832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.645105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.645753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:02.646057 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.647463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.650734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.651412 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.651697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.652232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.652930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:02.653233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.654669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.658775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 198/401] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:02.659413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.672101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 204/402] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:02.672858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.751998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:02.756586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a3da6b-a7b4-4749-9762-b042f6ded252 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.770681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd6d670-7462-42e5-98ae-15aa4a67b31c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.792861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da4d11-595f-4861-9c19-a2022cf8c072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.793466 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da4d11-595f-4861-9c19-a2022cf8c072 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.793741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da4d11-595f-4861-9c19-a2022cf8c072 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.794028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da4d11-595f-4861-9c19-a2022cf8c072 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.794660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5da4d11-595f-4861-9c19-a2022cf8c072 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.796121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5da4d11-595f-4861-9c19-a2022cf8c072 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.805891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.806514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.806795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.807102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.814148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.815639 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:02.827159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:02.827953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:02.828774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 205/403] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:02.829222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.856880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 199/404] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:43:02 2022] POST /identity/v3/users => generated 247 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:02.857740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da4d11-595f-4861-9c19-a2022cf8c072 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:02.867525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:02.955750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc32154-14d4-4b41-a792-4412a5b80355 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.977658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da4d11-595f-4861-9c19-a2022cf8c072 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:02.995961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:02.996585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:02.996853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:02.997118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:02.997767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:02.998063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:02.999515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.010413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:03.014066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 206/405] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:03.016001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.020731 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.021380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.021663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.021932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.031252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.032713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:03.045788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:03.046675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:03.047658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 200/406] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:02 2022] GET /identity/v3/users/neutron => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:03.048234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.123247 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bebcb1-6cf7-4818-a5fb-ceed0e9ad4cd None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:03.123951 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:03.158482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7e3e4e-03e5-418b-befa-2e009a76b48e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.159172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e3e4e-03e5-418b-befa-2e009a76b48e None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.159449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e3e4e-03e5-418b-befa-2e009a76b48e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.159725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e3e4e-03e5-418b-befa-2e009a76b48e None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/35fc6d90245c4939bc7171201eb883bd/roles/fa322a306cb442c79d45a314b16f1b85` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.161761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735e5e6c-0d76-4f1d-b8bb-dd986ae742d0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:03.174320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7e3e4e-03e5-418b-befa-2e009a76b48e None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=35fc6d90245c4939bc7171201eb883bd, role_id=fa322a306cb442c79d45a314b16f1b85)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.175792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7e3e4e-03e5-418b-befa-2e009a76b48e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.196866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.197475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.197753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.198025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.199274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:03.199578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.201025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.218746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 201/407] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:03.219597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.258996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 207/408] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:03 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/35fc6d90245c4939bc7171201eb883bd/roles/fa322a306cb442c79d45a314b16f1b85 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:03.258996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7e3e4e-03e5-418b-befa-2e009a76b48e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.310266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:03.364292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798d8fff-6fa0-4d64-a527-25f394637299 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:03.369065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7e3e4e-03e5-418b-befa-2e009a76b48e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:03.392958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:03.403790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.405087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.405535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.406054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.409806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.410571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.410992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.411385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.412304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:03.412757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.414341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.415528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.418103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:03.432060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:03.433532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:03.434402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:03.434402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:03.434402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:03.434402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:03.434402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:03.434402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:03.434402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:03.434402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:03.439014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 202/409] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:03.439519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 208/410] 10.222.0.20 () {56 vars in 1139 bytes} [Wed Oct 19 21:43:03 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:03.439866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.440929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.557813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11756ce7-6903-4ccf-a5dd-548d06d971ec None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:03.559568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd48927e-3ba2-4e2a-8f4c-281e49e4c945 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:03.568201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:03.595555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.595900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244012da-0d1e-4a35-bc06-c452743b02bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.596489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244012da-0d1e-4a35-bc06-c452743b02bc None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.596770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244012da-0d1e-4a35-bc06-c452743b02bc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.597048 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244012da-0d1e-4a35-bc06-c452743b02bc None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/9ebc359d9bf54edcb9634afc31cf188a/roles/fa322a306cb442c79d45a314b16f1b85` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.597349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.597894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.598515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.599443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:03.599957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.601612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.613081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244012da-0d1e-4a35-bc06-c452743b02bc None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=9ebc359d9bf54edcb9634afc31cf188a, role_id=fa322a306cb442c79d45a314b16f1b85)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.614561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244012da-0d1e-4a35-bc06-c452743b02bc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.615743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 209/411] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:03.616881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.687040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 203/412] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:03 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/9ebc359d9bf54edcb9634afc31cf188a/roles/fa322a306cb442c79d45a314b16f1b85 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:03.688484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244012da-0d1e-4a35-bc06-c452743b02bc None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.763331 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:03.772472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48be033-7b43-4dc3-b8b0-880fb759d7cc None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:03.851737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.852767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.853259 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.853729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.876269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.877767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:03.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:03.903547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:03.904394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:03.904394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:03.904394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:03.904394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 210/413] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:03.909409 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:03.911108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244012da-0d1e-4a35-bc06-c452743b02bc None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:03.932907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:03.962855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2328a62-67eb-4684-82e5-76f240fdd4f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:03.964572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2328a62-67eb-4684-82e5-76f240fdd4f0 None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:03.965017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2328a62-67eb-4684-82e5-76f240fdd4f0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:03.965429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2328a62-67eb-4684-82e5-76f240fdd4f0 None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/c065675e9a2f44f59168b9d574e289d5/roles/fa322a306cb442c79d45a314b16f1b85` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:03.984289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2328a62-67eb-4684-82e5-76f240fdd4f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=c065675e9a2f44f59168b9d574e289d5, role_id=fa322a306cb442c79d45a314b16f1b85)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:03.986282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2328a62-67eb-4684-82e5-76f240fdd4f0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:04.033494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01efff6f-d923-49e8-811c-6d1d5a353daf None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:04.070119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:04.070994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:04.071442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:04.071859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:04.072659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:04.073769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:04.075326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:04.089401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 211/414] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:04.097681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 204/415] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:03 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/c065675e9a2f44f59168b9d574e289d5/roles/fa322a306cb442c79d45a314b16f1b85 => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:04.098707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:04.194550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:04.217556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74bfcba9-31cf-4809-bda4-cc63b2dd10e4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:04.256748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:04.257740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:04.258163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:04.258623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:04.267198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:04.268899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:04.282392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:04.283497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:04.284604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:04.284604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:04.284604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:04.284604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:04.284604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:04.284604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:04.284604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:04.284604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:04.286354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 212/416] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:04.293445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2328a62-67eb-4684-82e5-76f240fdd4f0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:04.436217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:04.455883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2328a62-67eb-4684-82e5-76f240fdd4f0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:04.501623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dae527b-0dcb-451f-86fc-47dc6b939f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:04.502372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dae527b-0dcb-451f-86fc-47dc6b939f49 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:04.502748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dae527b-0dcb-451f-86fc-47dc6b939f49 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:04.503138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dae527b-0dcb-451f-86fc-47dc6b939f49 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:04.503928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dae527b-0dcb-451f-86fc-47dc6b939f49 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:04.504327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dae527b-0dcb-451f-86fc-47dc6b939f49 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:04.505911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dae527b-0dcb-451f-86fc-47dc6b939f49 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:04.523460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 205/417] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:04.529781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:04.671215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71e6629-fca3-4785-abf6-11a3e5c9e925 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:04.705365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d18072-59d2-4909-b754-382157358685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:04.706344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d18072-59d2-4909-b754-382157358685 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:04.706772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d18072-59d2-4909-b754-382157358685 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:04.707213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d18072-59d2-4909-b754-382157358685 None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/2f705dc9d2e541f1821dda4b5c7b7709/roles/fa322a306cb442c79d45a314b16f1b85` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:04.719288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:04.727241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d18072-59d2-4909-b754-382157358685 None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=2f705dc9d2e541f1821dda4b5c7b7709, role_id=fa322a306cb442c79d45a314b16f1b85)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:04.728700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d18072-59d2-4909-b754-382157358685 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:04.811917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 213/418] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:04 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/2f705dc9d2e541f1821dda4b5c7b7709/roles/fa322a306cb442c79d45a314b16f1b85 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:05.200814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfc94f7-45fa-4d4a-ae9b-fa84ff3612ff None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:05.201180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfc94f7-45fa-4d4a-ae9b-fa84ff3612ff None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:05.201382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfc94f7-45fa-4d4a-ae9b-fa84ff3612ff None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:05.202864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 206/419] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:05.208790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb8d8f-5a21-434a-856a-5db429ebe60b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:05.209294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb8d8f-5a21-434a-856a-5db429ebe60b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:05.209369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb8d8f-5a21-434a-856a-5db429ebe60b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:05.233918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-25fb8d8f-5a21-434a-856a-5db429ebe60b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:05.235169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-25fb8d8f-5a21-434a-856a-5db429ebe60b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:05.304614 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 214/420] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:05.728652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfc94f7-45fa-4d4a-ae9b-fa84ff3612ff None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:05.875564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfc94f7-45fa-4d4a-ae9b-fa84ff3612ff None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:05.924755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1001e8b-66bc-4be2-92c7-7075fb1dfb7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:05.925399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1001e8b-66bc-4be2-92c7-7075fb1dfb7b None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:05.925673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1001e8b-66bc-4be2-92c7-7075fb1dfb7b None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:05.925940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1001e8b-66bc-4be2-92c7-7075fb1dfb7b None admin] PATH_INFO: `/v3/roles/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:05.940857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1001e8b-66bc-4be2-92c7-7075fb1dfb7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-a1001e8b-66bc-4be2-92c7-7075fb1dfb7b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:05.946372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:05.947282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:05.948107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 207/421] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:05.956175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fb8d8f-5a21-434a-856a-5db429ebe60b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:06.055252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4969e604-a0d9-4a05-927c-f0dfd0bdbe26 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.055952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4969e604-a0d9-4a05-927c-f0dfd0bdbe26 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.056230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4969e604-a0d9-4a05-927c-f0dfd0bdbe26 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.058783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 208/422] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:06.063574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d36efb-90ef-45d0-950f-76f7a5b99f2d None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.063856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d36efb-90ef-45d0-950f-76f7a5b99f2d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.064136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d36efb-90ef-45d0-950f-76f7a5b99f2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.086845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:06.098314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-15d36efb-90ef-45d0-950f-76f7a5b99f2d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:06.099553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-15d36efb-90ef-45d0-950f-76f7a5b99f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:06.117931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fb8d8f-5a21-434a-856a-5db429ebe60b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:06.154422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:06.189376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503e74-9632-493a-8690-7916048b07ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:06.189994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503e74-9632-493a-8690-7916048b07ff None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.190269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503e74-9632-493a-8690-7916048b07ff None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.190544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29503e74-9632-493a-8690-7916048b07ff None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.191250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29503e74-9632-493a-8690-7916048b07ff None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:06.191554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29503e74-9632-493a-8690-7916048b07ff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:06.193168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29503e74-9632-493a-8690-7916048b07ff None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:06.211216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 215/423] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:06.216741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503e74-9632-493a-8690-7916048b07ff None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:06.236400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 209/424] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:43:06.356372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29503e74-9632-493a-8690-7916048b07ff None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:06.410257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be860303-7cd0-4019-92de-0e2d896b886f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:06.410899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be860303-7cd0-4019-92de-0e2d896b886f None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.411200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be860303-7cd0-4019-92de-0e2d896b886f None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.411480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be860303-7cd0-4019-92de-0e2d896b886f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.424991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be860303-7cd0-4019-92de-0e2d896b886f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:06.426452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be860303-7cd0-4019-92de-0e2d896b886f None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-be860303-7cd0-4019-92de-0e2d896b886f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:06.444678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:06.446601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:06.447725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:06.447725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:06.447725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:06.447725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 216/425] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:06.451398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d36efb-90ef-45d0-950f-76f7a5b99f2d None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:06.540529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806766a-40f8-481f-b978-5880e0dc5d91 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.540850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806766a-40f8-481f-b978-5880e0dc5d91 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.541070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b806766a-40f8-481f-b978-5880e0dc5d91 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.542463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 217/426] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:06.548276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4a8c36-3d6c-4f57-8a6c-16df80fdd718 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.548582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4a8c36-3d6c-4f57-8a6c-16df80fdd718 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.548823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4a8c36-3d6c-4f57-8a6c-16df80fdd718 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.577882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-7c4a8c36-3d6c-4f57-8a6c-16df80fdd718 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:06.579451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7c4a8c36-3d6c-4f57-8a6c-16df80fdd718 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:06.582418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d36efb-90ef-45d0-950f-76f7a5b99f2d None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:06.619699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:06.620271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.627041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.627041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.627041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:06.627041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:06.627041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:06.627041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:06.633216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:06.636816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 210/427] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:06.642449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:06.719170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 218/428] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:43:06.719170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49672af4-863a-4e7f-9363-00cfc3442284 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.719170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49672af4-863a-4e7f-9363-00cfc3442284 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.719170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49672af4-863a-4e7f-9363-00cfc3442284 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.719606 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 219/429] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:06.720115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49672af4-863a-4e7f-9363-00cfc3442284 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:06.753068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ceebb9-db88-49d4-bca5-5558ff372189 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:06.813274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b99aca1-bb31-4d9f-85ab-2509cf138fb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:06.814820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b99aca1-bb31-4d9f-85ab-2509cf138fb8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.815276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b99aca1-bb31-4d9f-85ab-2509cf138fb8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b99aca1-bb31-4d9f-85ab-2509cf138fb8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.827263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b99aca1-bb31-4d9f-85ab-2509cf138fb8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:06.829722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b99aca1-bb31-4d9f-85ab-2509cf138fb8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:06.838372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49672af4-863a-4e7f-9363-00cfc3442284 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-4b99aca1-bb31-4d9f-85ab-2509cf138fb8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:06.845621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:06.846716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 21:43:06.847841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:06.855444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 211/430] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:06 2022] GET /identity/v3/users/swift => generated 83 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:06.858221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a86982-e44b-4c4d-a999-ba9b3cca7a8d None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.858508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a86982-e44b-4c4d-a999-ba9b3cca7a8d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.858779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a86982-e44b-4c4d-a999-ba9b3cca7a8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.882689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-55a86982-e44b-4c4d-a999-ba9b3cca7a8d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:06.883863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-55a86982-e44b-4c4d-a999-ba9b3cca7a8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:06.886758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa78f22-0bff-4651-a436-ddc50ad8e526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:06.888509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa78f22-0bff-4651-a436-ddc50ad8e526 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:06.888992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa78f22-0bff-4651-a436-ddc50ad8e526 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:06.889429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa78f22-0bff-4651-a436-ddc50ad8e526 None admin] PATH_INFO: `/v3/roles/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:06.904329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa78f22-0bff-4651-a436-ddc50ad8e526 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-8aa78f22-0bff-4651-a436-ddc50ad8e526 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:06.911120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:06.912192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:06.913281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:06.916554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 220/431] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:06 2022] GET /identity/v3/roles/service => generated 85 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:06.918587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa78f22-0bff-4651-a436-ddc50ad8e526 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:06.947248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:06.956840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:06.965594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 212/432] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:06.966429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a86982-e44b-4c4d-a999-ba9b3cca7a8d None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.054744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa78f22-0bff-4651-a436-ddc50ad8e526 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.087180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a86982-e44b-4c4d-a999-ba9b3cca7a8d None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.095389 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.096102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.096473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.096839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.097663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:07.098061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.099733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.123562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 221/433] 10.222.0.20 () {56 vars in 1135 bytes} [Wed Oct 19 21:43:06 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:07.123562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.140937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.142225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.142807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.143357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.144403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] RBAC: Adding query filter params (name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:07.144855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.146820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.159544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 213/434] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:07.160597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.220776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:07.224393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:07.242499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-461be051-9edd-495b-aaba-6fd42f0463ea None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.285050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2e96bd-3268-40d8-8746-46297e6dff4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.286009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2e96bd-3268-40d8-8746-46297e6dff4c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.286449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2e96bd-3268-40d8-8746-46297e6dff4c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.286738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2e96bd-3268-40d8-8746-46297e6dff4c None admin] PATH_INFO: `/v3/roles/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.290733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af70705-1b3c-43ec-9cfb-f8e8d1a0f207 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.303345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2e96bd-3268-40d8-8746-46297e6dff4c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-1e2e96bd-3268-40d8-8746-46297e6dff4c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:07.308868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:07.309796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 222/435] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:07.312011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2e96bd-3268-40d8-8746-46297e6dff4c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.332093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.332733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.333016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.333294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.344266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.345745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:07.360928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:07.361804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:07.362632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:07.362632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:07.362632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:07.362632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 214/436] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:07.363180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.430244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2e96bd-3268-40d8-8746-46297e6dff4c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.463519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:07.464460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.464975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.465252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.465524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.466402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988902d7-be27-4018-ab53-815b381439a9 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.475356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.476812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.478825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:07.491523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:07.492449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:07.493515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:07.493515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:07.493515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:07.493515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 223/437] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:07.493515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.501830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe2ec1f-1b0e-4a84-8daa-3b9bcb1056c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.502463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe2ec1f-1b0e-4a84-8daa-3b9bcb1056c2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.502792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe2ec1f-1b0e-4a84-8daa-3b9bcb1056c2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.503106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe2ec1f-1b0e-4a84-8daa-3b9bcb1056c2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.513932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe2ec1f-1b0e-4a84-8daa-3b9bcb1056c2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-fbe2ec1f-1b0e-4a84-8daa-3b9bcb1056c2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:07.518660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:07.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:07.520547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 215/438] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:07.521183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe2ec1f-1b0e-4a84-8daa-3b9bcb1056c2 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.593884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669aa1a0-b8be-4553-8417-1eb65d34fb1b None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.618373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe2ec1f-1b0e-4a84-8daa-3b9bcb1056c2 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.626824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da098697-cca4-41d8-80f8-4b4bfc5f1c18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.627460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da098697-cca4-41d8-80f8-4b4bfc5f1c18 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.627738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da098697-cca4-41d8-80f8-4b4bfc5f1c18 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.628009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da098697-cca4-41d8-80f8-4b4bfc5f1c18 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.628672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da098697-cca4-41d8-80f8-4b4bfc5f1c18 None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:07.628989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da098697-cca4-41d8-80f8-4b4bfc5f1c18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.630360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da098697-cca4-41d8-80f8-4b4bfc5f1c18 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.638182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 224/439] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:07.639426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d01e8-e85d-406e-84cd-5ed9a9909f72 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.639726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d01e8-e85d-406e-84cd-5ed9a9909f72 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.640084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715d01e8-e85d-406e-84cd-5ed9a9909f72 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.642989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 225/440] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:07.642989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d01e8-e85d-406e-84cd-5ed9a9909f72 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.654355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.654985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.655265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.655542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.656199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:07.656507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.657892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.669506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 216/441] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:07.670296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.691296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:07.707365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:07.750666 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715d01e8-e85d-406e-84cd-5ed9a9909f72 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.770082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f4fc16-33f7-4ebd-ba10-67704343a227 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.786330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.786973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.787275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.787541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.788200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:07.788506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.789985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.803392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 226/442] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:07.804192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.807015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05430264-2ad1-4aa0-92d3-025495130b82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.807676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05430264-2ad1-4aa0-92d3-025495130b82 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.808084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05430264-2ad1-4aa0-92d3-025495130b82 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.808218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05430264-2ad1-4aa0-92d3-025495130b82 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.808883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05430264-2ad1-4aa0-92d3-025495130b82 None admin] RBAC: Adding query filter params (name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:07.809182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05430264-2ad1-4aa0-92d3-025495130b82 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.810584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05430264-2ad1-4aa0-92d3-025495130b82 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.818413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 217/443] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:07.819565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc6416-622d-4696-a8b3-4380dccbc64b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.819844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc6416-622d-4696-a8b3-4380dccbc64b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.820114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffcc6416-622d-4696-a8b3-4380dccbc64b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.837481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ffcc6416-622d-4696-a8b3-4380dccbc64b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:07.838772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ffcc6416-622d-4696-a8b3-4380dccbc64b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:07.904531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 218/444] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:07.905637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcc6416-622d-4696-a8b3-4380dccbc64b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:07.906496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94292178-42be-4a96-bdb2-7c0093bcf341 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:07.924176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:07.927735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:07.942396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:07.943058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:07.943336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:07.943612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:07.954385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:07.955908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:07.971859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:07.973019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:07.973847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:07.973847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:07.973847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:07.973847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 227/445] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:07.973847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.022637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffcc6416-622d-4696-a8b3-4380dccbc64b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.058333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.058971 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.059251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.059523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.067176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.069233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.076694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d15ea6-d127-4bb3-9312-799a563290e2 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:08.083112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:08.083935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 219/446] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:08.090318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.112587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.113235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.113512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.113782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] PATH_INFO: `/v3/users/nova` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.121366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.122813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:08.133706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:08.134593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 228/447] 10.222.0.20 () {56 vars in 1093 bytes} [Wed Oct 19 21:43:07 2022] GET /identity/v3/users/nova => generated 82 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:08.135506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.166010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:08.170385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:08.200323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d5e46bd-c844-471c-9324-3efc8a5b72bf None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.232994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.233607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.233886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.234154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.236468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b359905b-6a77-441f-8f0f-9d7ac21cd496 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.244674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.246991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:08.260777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:08.262217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:08.263327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:08.263327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:08.263327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:08.263327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 220/448] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:08.263327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.274170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.274802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.275109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.275381 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.276026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:08.276829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.278438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.290196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 229/449] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:08.291020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.364479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270662b7-a529-4be1-806a-51a430e772a5 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.378058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:08.388263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:08.390170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25588200-90c5-4471-bbd0-96529bb5759c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.398455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.399086 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.399363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.399631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.400319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:08.400994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.402382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.414117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 221/450] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:08.414709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.427304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.429637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.429637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.429637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.429637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:08.429637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.430907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.449684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 230/451] 10.222.0.20 () {56 vars in 1133 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:08.450293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.512697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33c7997-d931-4ae1-9e1f-543d0ccbb31f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.544514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b77f2de-056f-4855-8d61-afda056f3449 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.545796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c1c9b2-e431-4363-b3c1-637f70be7861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.546408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c1c9b2-e431-4363-b3c1-637f70be7861 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.546688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c1c9b2-e431-4363-b3c1-637f70be7861 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.546980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c1c9b2-e431-4363-b3c1-637f70be7861 None admin] PATH_INFO: `/v3/roles/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.556610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c1c9b2-e431-4363-b3c1-637f70be7861 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-51c1c9b2-e431-4363-b3c1-637f70be7861 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:08.561933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:08.563199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 222/452] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:08.564336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c1c9b2-e431-4363-b3c1-637f70be7861 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.584479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.585088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.585367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.585644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.586286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:08.586766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.588669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.599963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 231/453] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:08.600644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.608486 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:08.613326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:08.661088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c1c9b2-e431-4363-b3c1-637f70be7861 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.694292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.694916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.695225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.695526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] PATH_INFO: `/v3/users/glance` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.702705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62bf3b5-3710-4b66-bcd3-c3bd31cab0ee None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.703266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.704721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:08.716201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:08.717272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 223/454] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/users/glance => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:08.718349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.738546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeec2cd9-7dd1-4478-8c31-cff037c1af9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.739238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeec2cd9-7dd1-4478-8c31-cff037c1af9d None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.739517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeec2cd9-7dd1-4478-8c31-cff037c1af9d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.739791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeec2cd9-7dd1-4478-8c31-cff037c1af9d None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/b19bbdff6ae1428d8cfc7a33d4fafb49/roles/fa322a306cb442c79d45a314b16f1b85` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.753771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeec2cd9-7dd1-4478-8c31-cff037c1af9d None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=b19bbdff6ae1428d8cfc7a33d4fafb49, role_id=fa322a306cb442c79d45a314b16f1b85)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.755235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeec2cd9-7dd1-4478-8c31-cff037c1af9d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.817463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b96715-0989-467d-94cb-58091ee7648d None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.819706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:08.840594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 232/455] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:08 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/b19bbdff6ae1428d8cfc7a33d4fafb49/roles/fa322a306cb442c79d45a314b16f1b85 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:08.841661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeec2cd9-7dd1-4478-8c31-cff037c1af9d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:08.851773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35330605-6ca3-402d-b388-b78782d6cdfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.852423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35330605-6ca3-402d-b388-b78782d6cdfc None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.852709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35330605-6ca3-402d-b388-b78782d6cdfc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.852988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35330605-6ca3-402d-b388-b78782d6cdfc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.904346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:08.933301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35330605-6ca3-402d-b388-b78782d6cdfc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.934809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35330605-6ca3-402d-b388-b78782d6cdfc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:08.953098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeec2cd9-7dd1-4478-8c31-cff037c1af9d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:08.994694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:08.995978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:08.996396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:08.996743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:08.997580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:08.997954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:08.999510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.013010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 233/456] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:09.013010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.056913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 224/457] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:43:09.057745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35330605-6ca3-402d-b388-b78782d6cdfc None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.061723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:09.125384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285f9247-449d-4ca3-93ea-3d5ec7836fc8 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.142947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:09.160557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.161183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.161480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.161730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.170221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.171782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.171894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35330605-6ca3-402d-b388-b78782d6cdfc None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:09.183826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:09.184927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:09.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 234/458] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/users/cinder => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:09.190135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.219371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.220070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.220475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.220860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.221708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:09.222114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.223706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.241499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 225/459] 10.222.0.20 () {56 vars in 1137 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:09.242234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.299927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99860d8-0430-45b4-9360-2b854a9be70d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.303597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:09.337774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.338467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.338801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.339042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.342892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb72b10-a3b4-47ce-95a1-4ed7cb0b7915 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.350054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.351557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:09.366289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:09.367511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:09.368659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:09.368659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:09.368659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:09.368659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 235/460] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:09.368884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.382007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.382635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.382994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.383218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.394041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.395534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.401254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:09.409856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:09.410773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:09.412149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:09.412149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:09.412149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:09.412149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 226/461] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:09.412363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.480496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828dc87-1c7a-4d3d-84e7-debd457bbde9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.516322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.517273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.517758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.517862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-270e6fb6-c4c7-48e5-8efe-aa1754e022f2 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.518327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.519318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:09.519728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.521241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.532524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:09.544204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 236/462] 10.222.0.20 () {56 vars in 1137 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:09.544204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.557374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48516b4-c1c8-4bdb-80b6-e32ca7bbc3e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.558538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48516b4-c1c8-4bdb-80b6-e32ca7bbc3e9 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.558857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48516b4-c1c8-4bdb-80b6-e32ca7bbc3e9 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.559161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48516b4-c1c8-4bdb-80b6-e32ca7bbc3e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.632308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:09.633073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48516b4-c1c8-4bdb-80b6-e32ca7bbc3e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.635095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48516b4-c1c8-4bdb-80b6-e32ca7bbc3e9 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.646212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-095b11b8-bd58-4aca-a17b-0731b6ce7bb0 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.682335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.683503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.683787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.684066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.684882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:09.685188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.686870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.699252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 237/463] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:09.700067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.741931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 227/464] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:43:09.743070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48516b4-c1c8-4bdb-80b6-e32ca7bbc3e9 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.759827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:09.812638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b164bb-395b-47cd-98c5-39c265d4bc82 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.848819 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.849302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48516b4-c1c8-4bdb-80b6-e32ca7bbc3e9 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.850269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.850869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.851453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.852107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:09.852411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.853849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.871081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 238/465] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:09.871081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:09.881068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:09.889724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3105cd14-0967-461d-ae58-712ccbb5dfd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:09.889724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3105cd14-0967-461d-ae58-712ccbb5dfd4 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:09.890255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3105cd14-0967-461d-ae58-712ccbb5dfd4 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:09.890687 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3105cd14-0967-461d-ae58-712ccbb5dfd4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:09.962270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3105cd14-0967-461d-ae58-712ccbb5dfd4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:09.964567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3105cd14-0967-461d-ae58-712ccbb5dfd4 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:09.976430 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451af705-c3fb-41b7-a46c-bbfb39631a0c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:09.980222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:10.012783 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.013407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.013685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.013958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.021029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.022457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:10.034825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:10.035945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:10.036990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 239/466] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/users/neutron => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:10.037553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.073814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 228/467] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:43:10.074813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3105cd14-0967-461d-ae58-712ccbb5dfd4 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.116487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:10.153113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4cbf9f-0dc3-42b2-82d4-fa3f417fcdc9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.186110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.186442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3105cd14-0967-461d-ae58-712ccbb5dfd4 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.186759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.187060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.187334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.198617 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.200098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.202677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:10.214664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:10.222321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:10.223432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:10.223432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:10.223432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:10.223432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 240/468] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:10.223432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.230000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.230926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.231240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.231515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.238942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.240846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:10.254233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:10.255060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:10.255878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 229/469] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:10.256331 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.320720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf40e305-d834-4168-bc9f-29e594e37ecd None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.341105 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:10.356205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.356948 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.357322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.357663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.358457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:10.358878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.360404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.364271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54375b4d-8c49-43dc-96fa-063ae03dfbb2 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.378813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 241/470] 10.222.0.20 () {56 vars in 1139 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:10.379757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.404651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.405204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.405480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.405741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.416823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.418855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:10.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:10.434231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:10.437979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:10.437979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:10.437979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:10.437979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 230/471] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:10.437979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.465613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:10.492509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b9acf1-ce53-48a3-8fcd-76b822bd38fe None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.534774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.535604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.535924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.536271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.536995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:10.537373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.538818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.539649 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ded34-924f-4288-96cd-de595303d3e7 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.551449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 242/472] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:10.552369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.584102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.584737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.585016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.585287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.585985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:10.586290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.587773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.605292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 231/473] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:10.605292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.606626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:10.670152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04cd78b-dd53-4b2a-a84d-9442421269d5 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.693333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:10.708298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ee5d2-1c2c-4f30-8e48-c97512b506b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.708910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ee5d2-1c2c-4f30-8e48-c97512b506b6 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.709009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fe36c20-a6c8-48e7-a126-aab8344bbbfe None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.709186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ee5d2-1c2c-4f30-8e48-c97512b506b6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.709457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9ee5d2-1c2c-4f30-8e48-c97512b506b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.743946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.744550 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.744832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.745631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.746281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:10.746581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.748027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.760200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 232/474] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:10.765676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.782056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ee5d2-1c2c-4f30-8e48-c97512b506b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.784618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9ee5d2-1c2c-4f30-8e48-c97512b506b6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.827399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:10.870894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f61ae-531d-4dc0-a2a0-f877258efb35 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:10.899665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 243/475] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:43:10.899665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ee5d2-1c2c-4f30-8e48-c97512b506b6 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.912491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:10.913135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:10.913413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:10.913692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:10.921243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:10.922624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:10.934091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:10.934920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:10.935807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 233/476] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:10.936841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:10.946281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:11.012476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9ee5d2-1c2c-4f30-8e48-c97512b506b6 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.041799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88008118-16c5-4dd8-bc02-ba3f147cd668 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.050623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.051246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.051520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.051787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.052500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] RBAC: Adding query filter params (role.id=fa322a306cb442c79d45a314b16f1b85, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=35fc6d90245c4939bc7171201eb883bd) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:11.052802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.054226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.059814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 244/477] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/role_assignments?user.id=35fc6d90245c4939bc7171201eb883bd&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=fa322a306cb442c79d45a314b16f1b85 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:11.060577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.063082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:11.077615 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.078766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.079075 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.079780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.087203 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.089424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:11.102198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:11.103164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 234/478] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:11.109887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.179358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:11.187273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93a6998-5a70-4d9a-8240-a564187df914 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.222307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b69d985-c8f3-450e-99df-91c4994a40ca None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.228564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a340f6cb-ec83-4e95-a18d-9b293699f5f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.229191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a340f6cb-ec83-4e95-a18d-9b293699f5f9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.230122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a340f6cb-ec83-4e95-a18d-9b293699f5f9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.230390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a340f6cb-ec83-4e95-a18d-9b293699f5f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.243452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a340f6cb-ec83-4e95-a18d-9b293699f5f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.245055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a340f6cb-ec83-4e95-a18d-9b293699f5f9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-a340f6cb-ec83-4e95-a18d-9b293699f5f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:11.260159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:11.261459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:11.262456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:11.262456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:11.262456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:11.262456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 245/479] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:11.262715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660dc4f7-4093-45e0-a117-8dfad580d1eb None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.262873 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660dc4f7-4093-45e0-a117-8dfad580d1eb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.263239 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660dc4f7-4093-45e0-a117-8dfad580d1eb None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.264691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 246/480] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:11.265381 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660dc4f7-4093-45e0-a117-8dfad580d1eb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.268801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.270052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.270339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.270617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.271391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:11.271701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.273174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.288413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 235/481] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:11.291535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.330303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:11.379854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660dc4f7-4093-45e0-a117-8dfad580d1eb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.394267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfe72c3-f133-4131-baea-e57c8811b123 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.413057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73cd5e2-68f0-4ede-b032-8d95b1a490ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.413633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73cd5e2-68f0-4ede-b032-8d95b1a490ca None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.413908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73cd5e2-68f0-4ede-b032-8d95b1a490ca None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.414206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73cd5e2-68f0-4ede-b032-8d95b1a490ca None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.414898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73cd5e2-68f0-4ede-b032-8d95b1a490ca None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:11.415230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73cd5e2-68f0-4ede-b032-8d95b1a490ca None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.416668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73cd5e2-68f0-4ede-b032-8d95b1a490ca None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.423919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:11.440306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 247/482] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:11.440306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e95676-215e-4f79-ae4a-4f6603279403 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.440306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e95676-215e-4f79-ae4a-4f6603279403 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.440306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e95676-215e-4f79-ae4a-4f6603279403 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.440725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.440725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.440725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.440725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.441036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:11.441036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.441244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.450564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c7e95676-215e-4f79-ae4a-4f6603279403 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:11.451784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c7e95676-215e-4f79-ae4a-4f6603279403 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:11.458165 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 236/483] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:11.458165 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.528113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 248/484] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:11.528970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e95676-215e-4f79-ae4a-4f6603279403 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.566508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a71601-a2f7-46a8-b494-ee589b924f1e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.599321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:11.603884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.605043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.605323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.605588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.606370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] RBAC: Adding query filter params (role.id=fa322a306cb442c79d45a314b16f1b85, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=9ebc359d9bf54edcb9634afc31cf188a) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:11.606669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.608141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.619296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 237/485] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/role_assignments?user.id=9ebc359d9bf54edcb9634afc31cf188a&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=fa322a306cb442c79d45a314b16f1b85 => generated 608 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:11.619296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.685048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e95676-215e-4f79-ae4a-4f6603279403 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.710811 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:11.767469 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10255e6-af65-4f66-b2c6-1a4d9bbf3add None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.809135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.809910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.810196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.810646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.811589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] RBAC: Adding query filter params (role.id=fa322a306cb442c79d45a314b16f1b85, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=c065675e9a2f44f59168b9d574e289d5) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:11.811960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.812187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:11.812889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:11.813235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:11.813572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:11.813688 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.821785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 249/486] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/role_assignments?user.id=c065675e9a2f44f59168b9d574e289d5&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=fa322a306cb442c79d45a314b16f1b85 => generated 608 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:11.822653 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.832031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:11.833631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:11.847832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:11.848919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:11.849814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:11.849814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:11.849814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:11.849814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:11.849814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:11.849814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:11.849814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:11.849814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:11.851440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 238/487] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:11.858271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:11.973907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0854f43a-2d30-4cf3-9895-01801dfe414a None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:11.984388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:12.007165 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc353f5-46bc-4475-b2f8-e9ab3d8b3dd6 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:12.014655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5b33db-9703-4615-9f8d-c23f7b7e32ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:12.015553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5b33db-9703-4615-9f8d-c23f7b7e32ea None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:12.015901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5b33db-9703-4615-9f8d-c23f7b7e32ea None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:12.016286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5b33db-9703-4615-9f8d-c23f7b7e32ea None admin] PATH_INFO: `/v3/roles/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:12.025569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:12.030785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5b33db-9703-4615-9f8d-c23f7b7e32ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-cf5b33db-9703-4615-9f8d-c23f7b7e32ea None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:12.036698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:12.046904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 250/488] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:12.047844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5b33db-9703-4615-9f8d-c23f7b7e32ea None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:12.058721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:12.059500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:12.059852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:12.060195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:12.060997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:12.061377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:12.062920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:12.080968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 239/489] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:12.082353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:12.171348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5b33db-9703-4615-9f8d-c23f7b7e32ea None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:12.195365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8797ed4e-3cbf-40ca-8700-c17d12ca1867 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:12.218313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:12.219033 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:12.219377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:12.219715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:12.220445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:12.220811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:12.222242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:12.233294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 251/490] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:12.238388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:12.238610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:12.239316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:12.239674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:12.240071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:12.240885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] RBAC: Adding query filter params (role.id=fa322a306cb442c79d45a314b16f1b85, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=2f705dc9d2e541f1821dda4b5c7b7709) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:12.241351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:12.242893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:12.250637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 240/491] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:12 2022] GET /identity/v3/role_assignments?user.id=2f705dc9d2e541f1821dda4b5c7b7709&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=fa322a306cb442c79d45a314b16f1b85 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:12.295728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:12.368884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07746c9c-9195-4f65-9e9f-5ea92317b006 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:12.418475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:12.419182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:12.419458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:12.419726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:12.436708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:12.438184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:12.459776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:12.461720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:12.462848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:12.462848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:12.462848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:12.462848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 252/492] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:12.467114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:12.521196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:12.594442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd635d-50cf-435a-b2a4-67cc05fa2f6f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:12.639017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:12.639683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:12.640004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:12.640277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:12.640927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:12.641225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:12.642593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:12.655420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 241/493] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:12.660866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:12.752769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:12.776494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6e2dbc-38c6-40c2-b80a-615b1f94adc1 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:12.828600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:12.829238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:12.829512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:12.829777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] PATH_INFO: `/v3/users/swift` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:12.845081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:12.846562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:12.860629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:12.861672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:12.863332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 253/494] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:12 2022] GET /identity/v3/users/swift => generated 83 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:12.867727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:12.995849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:13.006468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735c553a-904b-4f09-a0ec-815fd5c8dd18 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:13.047391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:13.048001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:13.048245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:13.048454 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:13.049198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:13.049505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:13.050966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:13.069831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 242/495] 10.222.0.20 () {56 vars in 1135 bytes} [Wed Oct 19 21:43:12 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:13.075814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:13.262909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc2d1da-15e9-45a7-a550-25c6ea0ac8ec None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:13.301750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-346e51c6-425f-43e3-a916-d571d58fb788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:13.302274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346e51c6-425f-43e3-a916-d571d58fb788 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:13.302549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346e51c6-425f-43e3-a916-d571d58fb788 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:13.302818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-346e51c6-425f-43e3-a916-d571d58fb788 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:13.315211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:13.394552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346e51c6-425f-43e3-a916-d571d58fb788 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:13.396062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-346e51c6-425f-43e3-a916-d571d58fb788 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:13.512384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 254/496] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 19 21:43:13.517526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:13.525845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4197298c-7e12-45b8-83e0-3a6c81a9a334 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:13.526717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4197298c-7e12-45b8-83e0-3a6c81a9a334 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:13.527019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4197298c-7e12-45b8-83e0-3a6c81a9a334 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:13.528957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 255/497] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:13.534978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d6fc28-2ef6-42a2-83f9-77cf77d36bac None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:13.534978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d6fc28-2ef6-42a2-83f9-77cf77d36bac None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:13.534978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d6fc28-2ef6-42a2-83f9-77cf77d36bac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:13.561629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-83d6fc28-2ef6-42a2-83f9-77cf77d36bac None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:13.562745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-83d6fc28-2ef6-42a2-83f9-77cf77d36bac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:13.606257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:13.641229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 256/498] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:13.738600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ac6eac-09df-4f35-94d1-9303932d2a06 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:13.793145 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:13.793782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:13.794061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:13.794335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:13.815957 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:13.817364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:13.834602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:13.835518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:13.836952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:13.836952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:13.836952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:13.836952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 243/499] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:13.844037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d6fc28-2ef6-42a2-83f9-77cf77d36bac None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:13.920900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:13.925545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:13.966269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d6fc28-2ef6-42a2-83f9-77cf77d36bac None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:14.011067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:14.011067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.011067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.011067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.012751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:14.013057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:14.014436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:14.041402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 257/500] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:14.045934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:14.082427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:14.082725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60683628-f33e-4809-892d-be7b5c5a40ca None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:14.155398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e41a6-427d-4a49-b3c2-153d0fa9c4c2 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:14.164709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed75896-d250-4b6d-9fe0-c2fbfd198c1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:14.165348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed75896-d250-4b6d-9fe0-c2fbfd198c1c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.165626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed75896-d250-4b6d-9fe0-c2fbfd198c1c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.165898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed75896-d250-4b6d-9fe0-c2fbfd198c1c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.191694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:14.192325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.192604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.192872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.199555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed75896-d250-4b6d-9fe0-c2fbfd198c1c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:14.200498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:14.201963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-1ed75896-d250-4b6d-9fe0-c2fbfd198c1c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:14.205151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:14.206537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:14.207393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 244/501] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:14.218340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:14.220524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:14.221636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 258/502] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:14.222291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed75896-d250-4b6d-9fe0-c2fbfd198c1c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:14.228009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:14.330118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:14.345867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:14.346696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6969e94-7a27-493e-9197-82efe60cc013 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:14.394797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:14.395471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.395699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.395968 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.396658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:14.396963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:14.398903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed75896-d250-4b6d-9fe0-c2fbfd198c1c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:14.403569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:14.427173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 259/503] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:14.430511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:14.478058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:14.478829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.480062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.480342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.481304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:14.481611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:14.483536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:14.494123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 245/504] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:14.500487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:14.578139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34df817-8059-498a-a3ab-a9ea3ef76821 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:14.640791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa263cbc-df09-44cc-a961-baa93e54cc74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:14.641307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa263cbc-df09-44cc-a961-baa93e54cc74 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.641584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa263cbc-df09-44cc-a961-baa93e54cc74 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.641860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa263cbc-df09-44cc-a961-baa93e54cc74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.642591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa263cbc-df09-44cc-a961-baa93e54cc74 None admin] RBAC: Adding query filter params (role.id=fa322a306cb442c79d45a314b16f1b85, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=b19bbdff6ae1428d8cfc7a33d4fafb49) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:14.642899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa263cbc-df09-44cc-a961-baa93e54cc74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:14.644376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa263cbc-df09-44cc-a961-baa93e54cc74 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:14.655083 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 260/505] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:14 2022] GET /identity/v3/role_assignments?user.id=b19bbdff6ae1428d8cfc7a33d4fafb49&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=fa322a306cb442c79d45a314b16f1b85 => generated 608 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:14.661967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e7a40e-56c7-4b90-8384-db43bad41e10 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:14.693702 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:14.698478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b4a5f-8aa5-4ca2-bfe3-d516634d2c2b None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.698773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b4a5f-8aa5-4ca2-bfe3-d516634d2c2b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.699143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b4a5f-8aa5-4ca2-bfe3-d516634d2c2b None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.700402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 261/506] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:14.705626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:14.706286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.706566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.706731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305e9a48-99ed-4c36-83ab-8e30c96590d1 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.706830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.707035 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305e9a48-99ed-4c36-83ab-8e30c96590d1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.707294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305e9a48-99ed-4c36-83ab-8e30c96590d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.719988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:14.721797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:14.738865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:14.739879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:14.740735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:14.740735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:14.740735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:14.740900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-305e9a48-99ed-4c36-83ab-8e30c96590d1 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:14.741623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-305e9a48-99ed-4c36-83ab-8e30c96590d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:14.743176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 246/507] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:14.760062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:14.764996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:14.831494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 262/508] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:43:14.843378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad327dca-b419-415f-ba42-54fad74a010f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.843663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad327dca-b419-415f-ba42-54fad74a010f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.844005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad327dca-b419-415f-ba42-54fad74a010f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.845537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 263/509] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:14.851195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cac8b2-f683-4ad6-beec-d27ce9b35825 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.851664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cac8b2-f683-4ad6-beec-d27ce9b35825 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.851861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cac8b2-f683-4ad6-beec-d27ce9b35825 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.884631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a9cac8b2-f683-4ad6-beec-d27ce9b35825 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:14.887171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a9cac8b2-f683-4ad6-beec-d27ce9b35825 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:14.927867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1beeec3e-d161-4854-adae-4da218f392d4 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:14.964315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:14.965270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:14.965760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:14.966240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:14.967181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:14.967736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:14.969297 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:14.982066 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 247/510] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:14.988160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 264/511] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:43:14.988694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.031878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:15.100105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6063d2c-b7f1-4138-b705-3b9d903067fe None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.136549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.137171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.137444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.137710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] PATH_INFO: `/v3/users/nova` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.144901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.146373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.150936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cac8b2-f683-4ad6-beec-d27ce9b35825 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:15.159982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:15.160943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:15.162211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 248/512] 10.222.0.20 () {56 vars in 1093 bytes} [Wed Oct 19 21:43:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:15.166567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.242815 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:15.266228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cac8b2-f683-4ad6-beec-d27ce9b35825 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.291167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:15.297977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7ac8c-a089-4bde-8301-50c24121f101 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.305685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b6b0f9-3c45-4ad5-a6df-c98ba6e6b2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.306316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6b0f9-3c45-4ad5-a6df-c98ba6e6b2a2 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.306587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6b0f9-3c45-4ad5-a6df-c98ba6e6b2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.306854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b6b0f9-3c45-4ad5-a6df-c98ba6e6b2a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.317694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b6b0f9-3c45-4ad5-a6df-c98ba6e6b2a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.319310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b6b0f9-3c45-4ad5-a6df-c98ba6e6b2a2 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-65b6b0f9-3c45-4ad5-a6df-c98ba6e6b2a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:15.335297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:15.336257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:15.337761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:15.337761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:15.337761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:15.337953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.338052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 265/513] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:15.338052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c5be02-f687-4636-9ba7-1b4d12cb079c None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.338354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c5be02-f687-4636-9ba7-1b4d12cb079c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.338509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c5be02-f687-4636-9ba7-1b4d12cb079c None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.343016 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 266/514] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:15.343016 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c5be02-f687-4636-9ba7-1b4d12cb079c None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.343208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.343208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.343208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.343491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:15.343771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.345220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.365998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 249/515] 10.222.0.20 () {56 vars in 1133 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:15.367463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.458533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c5be02-f687-4636-9ba7-1b4d12cb079c None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.468480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1322020-3695-4c48-b3a5-9875f31225c6 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.489111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:15.502157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709b0671-ca8c-4df4-af13-671fd94750ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.502157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709b0671-ca8c-4df4-af13-671fd94750ba None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.502157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709b0671-ca8c-4df4-af13-671fd94750ba None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.502157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709b0671-ca8c-4df4-af13-671fd94750ba None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.502570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709b0671-ca8c-4df4-af13-671fd94750ba None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.502570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709b0671-ca8c-4df4-af13-671fd94750ba None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.503434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.504040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.504317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.504587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.505234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:15.505534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.506889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-709b0671-ca8c-4df4-af13-671fd94750ba None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:15.514300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:15.515387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:15.516420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:15.517359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 267/516] 10.222.0.20 () {56 vars in 1129 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:15.520239 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45480c22-2917-4df7-b5b9-91bc9027cab1 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.520623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45480c22-2917-4df7-b5b9-91bc9027cab1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.520994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45480c22-2917-4df7-b5b9-91bc9027cab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.525926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 250/517] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:15.525926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.540139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-45480c22-2917-4df7-b5b9-91bc9027cab1 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:15.541312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-45480c22-2917-4df7-b5b9-91bc9027cab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:15.546822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:15.619013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 268/518] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:15.619013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45480c22-2917-4df7-b5b9-91bc9027cab1 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.636298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88e1368-6015-428e-83bf-ab681b6f4960 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.673363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.673775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.674058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.674328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.685289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.686809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:15.702795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:15.703796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:15.704669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:15.704669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:15.704669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:15.708616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 251/519] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:15.709430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.745453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45480c22-2917-4df7-b5b9-91bc9027cab1 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.774378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:15.781699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:15.796599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.797495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.797949 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.798395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.799282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:15.799778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.801665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.808149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 269/520] 10.222.0.20 () {56 vars in 1124 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:15.809391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.821120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c9b96f-1a63-49db-b246-2034e2805183 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.857009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a92cd0-1aed-4f7f-b9e4-ad4dd2e8323e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.857663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a92cd0-1aed-4f7f-b9e4-ad4dd2e8323e None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.857939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a92cd0-1aed-4f7f-b9e4-ad4dd2e8323e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.858251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a92cd0-1aed-4f7f-b9e4-ad4dd2e8323e None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.859243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a92cd0-1aed-4f7f-b9e4-ad4dd2e8323e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.860369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a92cd0-1aed-4f7f-b9e4-ad4dd2e8323e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.920919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335251ce-36bc-4056-9a6d-95cd78f3c1b4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:15.927047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 252/521] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:43:15 2022] POST /identity/v3/users => generated 254 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:15.927876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a92cd0-1aed-4f7f-b9e4-ad4dd2e8323e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:15.965572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:15.966360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:15.966714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:15.967296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:15.968331 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:15.968777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:15.970569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:15.993297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 270/522] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:15.993297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:16.058972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a92cd0-1aed-4f7f-b9e4-ad4dd2e8323e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:16.059323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:16.085899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:16.098232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:16.099258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:16.099747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:16.100218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:16.101613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:16.102150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:16.103768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:16.115424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 253/523] 10.222.0.20 () {56 vars in 1124 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:16.115424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:16.125598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75686cc8-c869-4235-b8d7-9c8dbfc974ac None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:16.175114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:16.176108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:16.176529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:16.176945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] PATH_INFO: `/v3/services/network` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:16.177789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:16.180558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:16.192392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:16.193619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:16.194524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:16.197167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 271/524] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:15 2022] GET /identity/v3/services/network => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:16.203802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:16.269996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40df997-d055-4c20-9074-f0b0867b42af None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:16.337440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:16.337999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:16.338280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:16.338555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:16.350665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:16.352172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:16.353191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1234fb55-aa38-4798-8430-2db905ef6b43 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:16.372732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:16.373656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:16.374525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:16.374525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:16.374525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:16.374525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:16.374525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:16.374525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:16.374525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:16.374525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:16.377201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 254/525] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:16.380176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:16.383256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:16.404311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:16.404927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:16.408776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:16.408776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:16.408776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] RBAC: Adding query filter params (name=network) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:16.408776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:16.408776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:16.418800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 272/526] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:16 2022] GET /identity/v3/services?name=network => generated 124 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:16.425014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:16.460459 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:16.521767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c542a16c-3a32-464a-ae58-749bb0ed1f90 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:16.564082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:16.564698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:16.564976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:16.565247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:16.565949 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:16.572553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:16.574125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:16.598660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 255/527] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:16.605360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:16.638455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa026f7b-9d90-470c-87eb-954f0e49ce74 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:16.710282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d104b1b-29a9-4448-a42f-8804da48f0c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:16.711310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d104b1b-29a9-4448-a42f-8804da48f0c3 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:16.711937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d104b1b-29a9-4448-a42f-8804da48f0c3 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:16.712405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d104b1b-29a9-4448-a42f-8804da48f0c3 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:16.713227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d104b1b-29a9-4448-a42f-8804da48f0c3 None admin] RBAC: Adding query filter params (type=network) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:16.713784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d104b1b-29a9-4448-a42f-8804da48f0c3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:16.715387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d104b1b-29a9-4448-a42f-8804da48f0c3 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:16.727233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e1cff9b-a819-4cb4-b13f-bf98cffe43f1 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:16.730049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:16.730759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 273/528] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:16 2022] GET /identity/v3/services?type=network => generated 124 bytes in 306 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:16.774605 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db2b94d-1608-4be3-8e86-9928c2f22549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:16.775399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2b94d-1608-4be3-8e86-9928c2f22549 None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:16.775683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2b94d-1608-4be3-8e86-9928c2f22549 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:16.775958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db2b94d-1608-4be3-8e86-9928c2f22549 None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/35fc6d90245c4939bc7171201eb883bd/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:16.778194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db2b94d-1608-4be3-8e86-9928c2f22549 None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=35fc6d90245c4939bc7171201eb883bd, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:16.779701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db2b94d-1608-4be3-8e86-9928c2f22549 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:16.875522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 256/529] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:16 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/35fc6d90245c4939bc7171201eb883bd/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:17.211049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7af5248-65e4-464a-b82a-8c64f9d42dd1 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:17.211401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7af5248-65e4-464a-b82a-8c64f9d42dd1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:17.211599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7af5248-65e4-464a-b82a-8c64f9d42dd1 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:17.213149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 274/530] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:17.218462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b56456-2087-41db-96bc-37cdba12b3ca None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:17.218924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b56456-2087-41db-96bc-37cdba12b3ca None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:17.219041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b56456-2087-41db-96bc-37cdba12b3ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:17.242705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e4b56456-2087-41db-96bc-37cdba12b3ca None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:17.243959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e4b56456-2087-41db-96bc-37cdba12b3ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:17.313870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 257/531] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:17.693791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7af5248-65e4-464a-b82a-8c64f9d42dd1 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:17.743100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:17.838825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7af5248-65e4-464a-b82a-8c64f9d42dd1 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:17.886630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f846a4-ee3e-44e6-969b-f0a6184fc07e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:17.887626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f846a4-ee3e-44e6-969b-f0a6184fc07e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:17.888082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f846a4-ee3e-44e6-969b-f0a6184fc07e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:17.888529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f846a4-ee3e-44e6-969b-f0a6184fc07e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:17.903877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f846a4-ee3e-44e6-969b-f0a6184fc07e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-e4f846a4-ee3e-44e6-969b-f0a6184fc07e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:17.909361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:17.910250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:17.911100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:17.912968 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 275/532] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:17.918979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b56456-2087-41db-96bc-37cdba12b3ca None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:18.060077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:18.064296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b56456-2087-41db-96bc-37cdba12b3ca None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:18.104792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:18.105450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:18.105792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:18.106068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:18.106735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:18.107069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:18.108458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:18.123465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 258/533] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:18.132007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f846a4-ee3e-44e6-969b-f0a6184fc07e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:18.287218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:18.290759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f846a4-ee3e-44e6-969b-f0a6184fc07e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:18.324953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:18.325494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:18.325767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:18.326034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:18.336730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:18.338172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:18.352936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:18.353898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:18.355461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:18.355461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:18.355461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:18.355461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 276/534] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:18.359901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:18.525776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d81cc0-9c18-4fe8-aafb-327159907c52 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:18.574475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b899178c-8a1c-4da5-9a12-823a98c4bbb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:18.575170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b899178c-8a1c-4da5-9a12-823a98c4bbb1 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:18.575395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b899178c-8a1c-4da5-9a12-823a98c4bbb1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:18.575672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b899178c-8a1c-4da5-9a12-823a98c4bbb1 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:18.576324 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b899178c-8a1c-4da5-9a12-823a98c4bbb1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:18.576627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b899178c-8a1c-4da5-9a12-823a98c4bbb1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:18.578010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b899178c-8a1c-4da5-9a12-823a98c4bbb1 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:18.590982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 259/535] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:18.596015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:18.734064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c24644-ac53-4912-8711-5101d9f19a47 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:18.734418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c24644-ac53-4912-8711-5101d9f19a47 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:18.734615 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c24644-ac53-4912-8711-5101d9f19a47 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:18.736157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 260/536] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:18.741093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8383fd45-37b8-4a7f-a87f-20e27b6622d6 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:18.741616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8383fd45-37b8-4a7f-a87f-20e27b6622d6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:18.742087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8383fd45-37b8-4a7f-a87f-20e27b6622d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:18.745544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f07f5eea-2a64-4467-9d77-b1b614288121 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:18.760943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8383fd45-37b8-4a7f-a87f-20e27b6622d6 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:18.762441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8383fd45-37b8-4a7f-a87f-20e27b6622d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:18.778559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:18.805002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:18.805965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:18.806464 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:18.806926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] PATH_INFO: `/v3/users/swift` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:18.822488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:18.824314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:18.846458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:18.847438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 21:43:18.848349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:18.856985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 277/537] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:18 2022] GET /identity/v3/users/swift => generated 83 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:18.866575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:18.881813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:18.901851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 261/538] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 19 21:43:18.982593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1a9160-5f97-4c41-a238-7e5dc32640de None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:18.982980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1a9160-5f97-4c41-a238-7e5dc32640de None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:18.983184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1a9160-5f97-4c41-a238-7e5dc32640de None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:18.984799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 262/539] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:18.992204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afc1cc-3598-462e-9bc9-20c28b676a96 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:18.992488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afc1cc-3598-462e-9bc9-20c28b676a96 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:18.992767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7afc1cc-3598-462e-9bc9-20c28b676a96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.002933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35200ef3-2046-4716-9113-917c9a1741e9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:19.028918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b7afc1cc-3598-462e-9bc9-20c28b676a96 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:19.030112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b7afc1cc-3598-462e-9bc9-20c28b676a96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:19.040800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:19.041717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.042743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.043305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.044232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:19.044730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:19.046485 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:19.066575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 278/540] 10.222.0.20 () {56 vars in 1135 bytes} [Wed Oct 19 21:43:18 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:19.075987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.144713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:19.146481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 263/541] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:19.180091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68329b5e-0e24-4ab8-8664-c2f6fa4f17e7 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:19.216396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:19.217416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.217892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.218379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.230482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:19.232387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:19.249742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:19.250740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:19.251715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:19.251715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:19.251715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:19.253460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 279/542] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:19.258780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afc1cc-3598-462e-9bc9-20c28b676a96 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.305552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.376631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:19.396472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7afc1cc-3598-462e-9bc9-20c28b676a96 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:19.409011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f4968f0-04ae-4ec4-8c25-841c7e77d078 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:19.429680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:19.435744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5ba358-a993-4012-9870-d470e94ad008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:19.436362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ba358-a993-4012-9870-d470e94ad008 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.436643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ba358-a993-4012-9870-d470e94ad008 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.436916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5ba358-a993-4012-9870-d470e94ad008 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.441721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5ba358-a993-4012-9870-d470e94ad008 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:19.441721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5ba358-a993-4012-9870-d470e94ad008 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:19.441721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5ba358-a993-4012-9870-d470e94ad008 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:19.445274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d792ad-d167-4ffe-b4cf-ce82decf1447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:19.446162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d792ad-d167-4ffe-b4cf-ce82decf1447 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.446625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d792ad-d167-4ffe-b4cf-ce82decf1447 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.447096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d792ad-d167-4ffe-b4cf-ce82decf1447 None admin] PATH_INFO: `/v3/roles/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.456882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 264/543] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:19.458786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1d3fe2-21b6-4695-aecc-e04bf711c2f0 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.459103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1d3fe2-21b6-4695-aecc-e04bf711c2f0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.459397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1d3fe2-21b6-4695-aecc-e04bf711c2f0 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.459570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d792ad-d167-4ffe-b4cf-ce82decf1447 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:19.461383 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 265/544] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:19.462245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1d3fe2-21b6-4695-aecc-e04bf711c2f0 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-81d792ad-d167-4ffe-b4cf-ce82decf1447 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:19.465343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:19.466249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:19.467119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:19.468193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 280/545] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:19.472940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ed35f-70cc-40f6-9f11-72b0fb6b4519 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.472940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ed35f-70cc-40f6-9f11-72b0fb6b4519 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.472940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778ed35f-70cc-40f6-9f11-72b0fb6b4519 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.503308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-778ed35f-70cc-40f6-9f11-72b0fb6b4519 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:19.504846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-778ed35f-70cc-40f6-9f11-72b0fb6b4519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:19.572486 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1d3fe2-21b6-4695-aecc-e04bf711c2f0 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:19.591403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 281/546] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:19.592191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778ed35f-70cc-40f6-9f11-72b0fb6b4519 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.627524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:19.628158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.628434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.628701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.643021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:19.645561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:19.649631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:19.663374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:19.664278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:19.665143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:19.665143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:19.665143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:19.665143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:19.665143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:19.665143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:19.665143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:19.665143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:19.672887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 266/547] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:19.672887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.712988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778ed35f-70cc-40f6-9f11-72b0fb6b4519 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:19.724780 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:19.750174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d435fb85-67f5-4659-a178-51398904436b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:19.750810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d435fb85-67f5-4659-a178-51398904436b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.751067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d435fb85-67f5-4659-a178-51398904436b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.751353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d435fb85-67f5-4659-a178-51398904436b None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.752015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d435fb85-67f5-4659-a178-51398904436b None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:19.752315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d435fb85-67f5-4659-a178-51398904436b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:19.753878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d435fb85-67f5-4659-a178-51398904436b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:19.768420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 282/548] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:19 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:19.768420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d435fb85-67f5-4659-a178-51398904436b None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.784865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231a39b9-7272-47f9-b301-cfb65e1282c2 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:19.822537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc412dc-2725-4994-b58f-8f8d281eb4c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:19.823192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc412dc-2725-4994-b58f-8f8d281eb4c9 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.823472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc412dc-2725-4994-b58f-8f8d281eb4c9 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.823743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc412dc-2725-4994-b58f-8f8d281eb4c9 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.824397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc412dc-2725-4994-b58f-8f8d281eb4c9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:19.825758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc412dc-2725-4994-b58f-8f8d281eb4c9 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:19.875052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d435fb85-67f5-4659-a178-51398904436b None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:19.907361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c09352-353b-419a-8141-9403ad91481d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:19.907900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c09352-353b-419a-8141-9403ad91481d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.908176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c09352-353b-419a-8141-9403ad91481d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.908446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c09352-353b-419a-8141-9403ad91481d None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.909131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c09352-353b-419a-8141-9403ad91481d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:19.909433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c09352-353b-419a-8141-9403ad91481d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:19.910854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c09352-353b-419a-8141-9403ad91481d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:19.920748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 267/549] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:43:19 2022] POST /identity/v3/services => generated 251 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:19.922070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a532ba33-b2aa-4f47-aeff-defcfb079e18 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:19.922362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a532ba33-b2aa-4f47-aeff-defcfb079e18 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:19.922636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a532ba33-b2aa-4f47-aeff-defcfb079e18 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:19.923300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 283/550] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:19.928594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c09352-353b-419a-8141-9403ad91481d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.928719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 268/551] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:19 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:19.929376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a532ba33-b2aa-4f47-aeff-defcfb079e18 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:19.964445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:20.002585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:20.042436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c09352-353b-419a-8141-9403ad91481d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:20.055861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a532ba33-b2aa-4f47-aeff-defcfb079e18 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:20.082122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bd2262-e7aa-4af1-8b40-f439bec82832 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:20.082768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bd2262-e7aa-4af1-8b40-f439bec82832 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.083073 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bd2262-e7aa-4af1-8b40-f439bec82832 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.083347 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bd2262-e7aa-4af1-8b40-f439bec82832 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.094613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d00b99d-061a-4000-85a1-91baab34e418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:20.095127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bd2262-e7aa-4af1-8b40-f439bec82832 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:20.095541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00b99d-061a-4000-85a1-91baab34e418 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.095824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00b99d-061a-4000-85a1-91baab34e418 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.096099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d00b99d-061a-4000-85a1-91baab34e418 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.096565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bd2262-e7aa-4af1-8b40-f439bec82832 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:20.107996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d00b99d-061a-4000-85a1-91baab34e418 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-43bd2262-e7aa-4af1-8b40-f439bec82832 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:20.111439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:20.112354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:20.113621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:20.113621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:20.113621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:20.113621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 284/552] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-4d00b99d-061a-4000-85a1-91baab34e418 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:20.114134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:20.115029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:20.115855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:20.116491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f1f9b1-332a-4e54-9b32-7ed6c102f363 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.116491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f1f9b1-332a-4e54-9b32-7ed6c102f363 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.116491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f1f9b1-332a-4e54-9b32-7ed6c102f363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.116666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 269/553] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:20.116666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d00b99d-061a-4000-85a1-91baab34e418 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:20.133097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-59f1f9b1-332a-4e54-9b32-7ed6c102f363 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:20.134925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-59f1f9b1-332a-4e54-9b32-7ed6c102f363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:20.206733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 285/554] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:20 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 19 21:43:20.207556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f1f9b1-332a-4e54-9b32-7ed6c102f363 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:20.225121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:20.238748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d00b99d-061a-4000-85a1-91baab34e418 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:20.263684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:20.287100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dc6d82-c471-4d2e-bfb9-32ebbe513767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:20.288026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dc6d82-c471-4d2e-bfb9-32ebbe513767 None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.288305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dc6d82-c471-4d2e-bfb9-32ebbe513767 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.288596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dc6d82-c471-4d2e-bfb9-32ebbe513767 None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/b19bbdff6ae1428d8cfc7a33d4fafb49/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.292628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11dc6d82-c471-4d2e-bfb9-32ebbe513767 None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=b19bbdff6ae1428d8cfc7a33d4fafb49, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:20.294082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11dc6d82-c471-4d2e-bfb9-32ebbe513767 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:20.321704 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f1f9b1-332a-4e54-9b32-7ed6c102f363 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:20.358516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:20.359204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.359483 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.359756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.360403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:20.360704 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:20.362141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:20.368561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 270/555] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:20 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/b19bbdff6ae1428d8cfc7a33d4fafb49/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:20.368561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dc6d82-c471-4d2e-bfb9-32ebbe513767 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:20.391428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 286/556] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:20.394047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:20.484577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:20.531296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beacc2ac-e328-449c-b859-1124de4229e3 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:20.574700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11dc6d82-c471-4d2e-bfb9-32ebbe513767 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:20.577303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:20.577953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.578232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.578507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.587790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:20.590245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:20.604787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:20.605896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 287/557] 10.222.0.20 () {56 vars in 1117 bytes} [Wed Oct 19 21:43:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:20.607195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:20.617257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:20.618317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.618591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.618859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.619976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:20.620286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:20.622035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:20.631125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 271/558] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:20.631965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:20.689317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:20.721923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0e2153-0cf7-46c7-a151-e2a75ac9c035 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:20.760053 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189b92eb-ab56-4188-9857-25bbcdfab929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:20.760701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b92eb-ab56-4188-9857-25bbcdfab929 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.766977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b92eb-ab56-4188-9857-25bbcdfab929 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.766977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189b92eb-ab56-4188-9857-25bbcdfab929 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.766977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189b92eb-ab56-4188-9857-25bbcdfab929 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:20.766977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189b92eb-ab56-4188-9857-25bbcdfab929 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:20.767778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:20.769283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fadf8f4-4461-44da-a8a6-20def359127a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:20.805610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:20.806288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:20.806571 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:20.806843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:20.808177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:20.808488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:20.809911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:20.827872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 272/559] 10.222.0.20 () {56 vars in 1149 bytes} [Wed Oct 19 21:43:20 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:20.828873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:20.835234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 288/560] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:43:20 2022] POST /identity/v3/services => generated 240 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:20.835234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189b92eb-ab56-4188-9857-25bbcdfab929 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:21.036916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:21.046899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f180bc3-93e6-4254-9101-a669a704c45e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:21.059591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189b92eb-ab56-4188-9857-25bbcdfab929 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:21.087684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:21.088647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:21.089113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:21.089673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:21.100862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:21.102488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:21.102902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:21.103711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:21.103711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:21.103711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:21.115076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:21.116543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:21.122026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:21.123017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:21.123908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:21.123908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:21.123908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:21.125719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 273/561] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:21.132971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:21.133901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:21.134746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:21.134746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:21.134746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:21.134746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 289/562] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 302 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:21.135394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:21.156030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:21.231455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:21.247421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c4d858-7396-447c-a9d8-e1cf39a6dcb3 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:21.285185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:21.285861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:21.289073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:21.289073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:21.289073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:21.289073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:21.289073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:21.302240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 274/563] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:21.309148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:21.335486 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:21.349988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473f02e5-093f-4126-a722-a8fe8f186992 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:21.420124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:21.420736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:21.421012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:21.421283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:21.421952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:21.422263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:21.423715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:21.450868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 290/564] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:21.457028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:21.521176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567ca2d0-92b1-4c14-8cd9-35f770469607 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:21.578374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96e8a3a-dc6d-4717-95ce-ea06ea60b6b4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:21.579841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:21.619272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:21.619919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:21.621195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:21.621873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:21.622149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:21.622412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:21.623146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:21.623421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:21.636452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:21.637889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:21.638945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:21.640398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:21.659011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:21.660000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:21.660965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 275/565] 10.222.0.20 () {56 vars in 1093 bytes} [Wed Oct 19 21:43:21 2022] GET /identity/v3/users/nova => generated 82 bytes in 352 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:21.665111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:21.667548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:21.669576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:21.669576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:21.669576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:21.669576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:21.669576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:21.669576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:21.669576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:21.669576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:21.670531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:21.670864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 291/566] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:21.680357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:21.805355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:21.840287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf36c5f5-a2e4-42fe-9178-03f77e9005f2 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:21.857296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112a5ed8-bdc0-4b02-ae2a-2d2237b8ceda None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:21.906769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:21.907414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:21.907699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:21.907976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:21.908728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:21.909037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:21.910513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:21.927981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:21.928614 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:21.928893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:21.929163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:21.929875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:21.930178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:21.931658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:21.946499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 292/568] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:21 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:21.946688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 276/568] 10.222.0.20 () {56 vars in 1133 bytes} [Wed Oct 19 21:43:21 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:21.952453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:21.957031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:22.000561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:22.088757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0f45d7-d011-448b-b3f7-406c627b19d7 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:22.127227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6d230f-8ca6-4f4d-981d-d93e347d72f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:22.128228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d230f-8ca6-4f4d-981d-d93e347d72f9 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:22.128840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d230f-8ca6-4f4d-981d-d93e347d72f9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:22.129335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6d230f-8ca6-4f4d-981d-d93e347d72f9 None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/eba069a7652d4c26a6a35a73cb2be767/roles/fa322a306cb442c79d45a314b16f1b85` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:22.145106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6d230f-8ca6-4f4d-981d-d93e347d72f9 None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=eba069a7652d4c26a6a35a73cb2be767, role_id=fa322a306cb442c79d45a314b16f1b85)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:22.146815 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6d230f-8ca6-4f4d-981d-d93e347d72f9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:22.163440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec18bae-8e1d-49c9-9939-c505fea4be99 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:22.190509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:22.200166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcde059-f56d-4236-b3aa-cf841e68f3ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:22.201116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcde059-f56d-4236-b3aa-cf841e68f3ca None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:22.201614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcde059-f56d-4236-b3aa-cf841e68f3ca None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:22.202086 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcde059-f56d-4236-b3aa-cf841e68f3ca None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:22.218846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 293/569] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:21 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/eba069a7652d4c26a6a35a73cb2be767/roles/fa322a306cb442c79d45a314b16f1b85 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:22.230094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674ba47-a7a9-4180-b7a2-0fcb608f3902 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:22.230094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674ba47-a7a9-4180-b7a2-0fcb608f3902 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:22.230094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674ba47-a7a9-4180-b7a2-0fcb608f3902 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:22.230094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 294/570] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:22.232599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480100c2-083e-4f28-85ec-ec9a22516c80 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:22.233119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480100c2-083e-4f28-85ec-ec9a22516c80 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:22.233644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480100c2-083e-4f28-85ec-ec9a22516c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:22.258778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-480100c2-083e-4f28-85ec-ec9a22516c80 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:22.260113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-480100c2-083e-4f28-85ec-ec9a22516c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:22.340735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcde059-f56d-4236-b3aa-cf841e68f3ca None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:22.342214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcde059-f56d-4236-b3aa-cf841e68f3ca None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:22.366171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 295/571] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:22.479777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 277/572] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:21 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 528 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:43:22.485290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480100c2-083e-4f28-85ec-ec9a22516c80 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:22.487958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:22.591798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480100c2-083e-4f28-85ec-ec9a22516c80 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:22.628068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:22.628691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:22.628966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:22.629234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:22.639506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:22.640963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:22.656531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:22.657503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:22.658375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:22.658375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:22.658375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:22.658375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 296/573] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:22.663681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcde059-f56d-4236-b3aa-cf841e68f3ca None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:22.686564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:22.692071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:22.806941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcde059-f56d-4236-b3aa-cf841e68f3ca None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:22.812829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:22.849135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:22.849751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:22.850029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:22.850301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:22.850975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:22.851278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:22.852910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:22.870780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 278/574] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:22.878933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:22.920391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd76385-6fe1-4711-8ed1-52821d67a95a None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:22.964647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb357064-07f4-4932-b891-f32535bdbf5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:22.965816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb357064-07f4-4932-b891-f32535bdbf5f None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:22.966313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb357064-07f4-4932-b891-f32535bdbf5f None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:22.966832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb357064-07f4-4932-b891-f32535bdbf5f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:22.967633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb357064-07f4-4932-b891-f32535bdbf5f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:22.969253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb357064-07f4-4932-b891-f32535bdbf5f None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-bb357064-07f4-4932-b891-f32535bdbf5f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:22.981794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:22.982822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:22.983681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:22.986217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 297/575] 10.222.0.20 () {56 vars in 1129 bytes} [Wed Oct 19 21:43:22 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:22.987584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da11a9a2-5039-4777-b2c1-61fd69857b3f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:22.988492 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da11a9a2-5039-4777-b2c1-61fd69857b3f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:22.988955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da11a9a2-5039-4777-b2c1-61fd69857b3f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:22.990650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 298/576] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:22.991703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da11a9a2-5039-4777-b2c1-61fd69857b3f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:23.026439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:23.041592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf17dfb2-b65f-4e3a-aff7-d7a3359224c0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:23.084009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a1ea63-e14c-4d7f-bb5a-e583c4e07dc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:23.084945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1ea63-e14c-4d7f-bb5a-e583c4e07dc8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.085458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1ea63-e14c-4d7f-bb5a-e583c4e07dc8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.085942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a1ea63-e14c-4d7f-bb5a-e583c4e07dc8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.095943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a1ea63-e14c-4d7f-bb5a-e583c4e07dc8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:23.098352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a1ea63-e14c-4d7f-bb5a-e583c4e07dc8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:23.114471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da11a9a2-5039-4777-b2c1-61fd69857b3f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-b8a1ea63-e14c-4d7f-bb5a-e583c4e07dc8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:23.114686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:23.115582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:23.116614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:23.116614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:23.116614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:23.116614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:23.116614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:23.116614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:23.116614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:23.116614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:23.120728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 279/577] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:23.122069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed0d492-2b2f-471c-b170-3496dae68950 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.122528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed0d492-2b2f-471c-b170-3496dae68950 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.123018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed0d492-2b2f-471c-b170-3496dae68950 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.136887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:23.148177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-bed0d492-2b2f-471c-b170-3496dae68950 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:23.150672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-bed0d492-2b2f-471c-b170-3496dae68950 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:23.158024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:23.158691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.159002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.159290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.160042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:23.160350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:23.161740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:23.169215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 299/578] 10.222.0.20 () {56 vars in 1124 bytes} [Wed Oct 19 21:43:22 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:23.175082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:23.238009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 280/579] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:23 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:23.239295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed0d492-2b2f-471c-b170-3496dae68950 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:23.315818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a7e8ff-a13e-49a8-958d-d9c3efc1f1f0 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:23.322869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:23.346909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed0d492-2b2f-471c-b170-3496dae68950 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:23.362668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:23.364831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.365968 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.368055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.369748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:23.371656 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:23.373528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:23.402930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 300/580] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:23.407002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:23.414261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:23.415290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.419471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.420062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.421053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:23.421570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:23.423431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:23.434988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 281/581] 10.222.0.20 () {56 vars in 1124 bytes} [Wed Oct 19 21:43:23 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:23.436696 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:23.500467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:23.570643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b21795-f832-4425-b0a0-900d32af64d7 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:23.574260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc4d676-0497-4796-a147-946c822a2345 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:23.608892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4ff373-463b-4933-bc2d-0a3e8f9a8b1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:23.609491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ff373-463b-4933-bc2d-0a3e8f9a8b1b None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.609777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ff373-463b-4933-bc2d-0a3e8f9a8b1b None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.610152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ff373-463b-4933-bc2d-0a3e8f9a8b1b None admin] PATH_INFO: `/v3/endpoints` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.610984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4ff373-463b-4933-bc2d-0a3e8f9a8b1b None admin] RBAC: Adding query filter params (interface=public, service_id=fd39f0be3eab4f20a1185b97f99fe026, region_id=RegionOne) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:23.611309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4ff373-463b-4933-bc2d-0a3e8f9a8b1b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:23.612793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4ff373-463b-4933-bc2d-0a3e8f9a8b1b None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:23.613919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:23.615081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.615587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.616056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.617723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=35fc6d90245c4939bc7171201eb883bd) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:23.618141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:23.619693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:23.620697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 282/582] 10.222.0.20 () {56 vars in 1251 bytes} [Wed Oct 19 21:43:23 2022] GET /identity/v3/endpoints?service_id=fd39f0be3eab4f20a1185b97f99fe026&interface=public®ion_id=RegionOne => generated 194 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:23.622001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05114a44-2da4-4893-8253-becfb539bee2 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.622292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05114a44-2da4-4893-8253-becfb539bee2 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.622568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05114a44-2da4-4893-8253-becfb539bee2 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.626700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 301/583] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:23 2022] GET /identity/v3/role_assignments?user.id=35fc6d90245c4939bc7171201eb883bd&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 608 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:23.628018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 283/584] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:23 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:23.628153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:23.628796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05114a44-2da4-4893-8253-becfb539bee2 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:23.633070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:23.750656 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f9c7c0-2786-4e25-a37c-3ed52cfdfe88 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:23.758421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:23.779942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05114a44-2da4-4893-8253-becfb539bee2 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:23.793580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5e4e85-2830-42e6-b290-33a35e1951f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:23.794465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e4e85-2830-42e6-b290-33a35e1951f7 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.795002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e4e85-2830-42e6-b290-33a35e1951f7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.795778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5e4e85-2830-42e6-b290-33a35e1951f7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.807314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5e4e85-2830-42e6-b290-33a35e1951f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-cc5e4e85-2830-42e6-b290-33a35e1951f7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:23.813641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:23.814781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:23.815674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:23.818176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 302/585] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:23.819619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978b25a-f42d-45ad-9386-586113cb308c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.820099 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978b25a-f42d-45ad-9386-586113cb308c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.820546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e978b25a-f42d-45ad-9386-586113cb308c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.827803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53220e6a-f489-498a-961b-3f67ace49bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:23.827803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53220e6a-f489-498a-961b-3f67ace49bf3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:23.828040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53220e6a-f489-498a-961b-3f67ace49bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:23.828435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53220e6a-f489-498a-961b-3f67ace49bf3 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:23.829190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53220e6a-f489-498a-961b-3f67ace49bf3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:23.831456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53220e6a-f489-498a-961b-3f67ace49bf3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:23.837277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 284/586] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:43:23 2022] GET /identity/v3/services => generated 772 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:23.839166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53220e6a-f489-498a-961b-3f67ace49bf3 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:23.841911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e978b25a-f42d-45ad-9386-586113cb308c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:23.843549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e978b25a-f42d-45ad-9386-586113cb308c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:23.921422 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:23.931468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 303/587] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:23 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:23.962086 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53220e6a-f489-498a-961b-3f67ace49bf3 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:24.000948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:24.001947 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:24.002232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:24.002565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:24.003298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:24.003611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:24.005138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:24.015755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 285/588] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:24.021047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e978b25a-f42d-45ad-9386-586113cb308c None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:24.118433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:24.125374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e978b25a-f42d-45ad-9386-586113cb308c None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:24.164673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:24.165395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:24.165680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:24.165959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:24.176306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:24.178358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:24.193865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:24.194812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:24.195725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:24.195725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:24.195725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:24.195725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 304/589] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:24.203197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:24.266443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:24.362275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8551561b-4533-4739-be82-8eb4f2676768 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:24.392981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:24.398689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:24.399333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:24.399681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:24.399959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:24.400616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:24.400987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:24.402405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:24.415801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 286/590] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:24.422655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:24.465573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364050c0-19ea-4f91-924c-32b925c641ac None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:24.507387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:24.508019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:24.511019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:24.511019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] PATH_INFO: `/v3/services/network` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:24.511019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:24.511807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:24.530642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:24.531751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:24.532596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:24.534698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 305/591] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:24 2022] GET /identity/v3/services/network => generated 88 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:24.543656 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:24.563313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:24.664083 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca1ca9a-f457-4841-947e-f421734085b8 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:24.669118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e82a81-7f97-4ce1-8811-b760049072f8 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:24.704784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:24.705453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:24.705736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:24.706010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:24.706648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] RBAC: Adding query filter params (name=network) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:24.706965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:24.708775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:24.715072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 306/592] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:24 2022] GET /identity/v3/services?name=network => generated 124 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:24.721435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:24.757633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:24.758275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:24.758550 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:24.758813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:24.767243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:24.768718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:24.782309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:24.785105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:24.786740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 287/593] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:24 2022] GET /identity/v3/users/swift => generated 83 bytes in 361 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:24.795003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:24.821911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:24.863172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384aaa23-3cde-4a72-99eb-8d27f7cf8a17 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:24.883263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:24.907455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38615e8f-3b61-4d75-83ec-b46b1666d9bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:24.908433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38615e8f-3b61-4d75-83ec-b46b1666d9bb None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:24.909468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38615e8f-3b61-4d75-83ec-b46b1666d9bb None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:24.910091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38615e8f-3b61-4d75-83ec-b46b1666d9bb None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:24.913448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38615e8f-3b61-4d75-83ec-b46b1666d9bb None admin] RBAC: Adding query filter params (type=network) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:24.914804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38615e8f-3b61-4d75-83ec-b46b1666d9bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:24.916413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38615e8f-3b61-4d75-83ec-b46b1666d9bb None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:24.926489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 307/594] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:24 2022] GET /identity/v3/services?type=network => generated 350 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:24.928186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c3a6f4-a77a-44b0-8077-731f9baabc6b None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:24.928683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c3a6f4-a77a-44b0-8077-731f9baabc6b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:24.929166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c3a6f4-a77a-44b0-8077-731f9baabc6b None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:24.930825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 308/595] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:24.939874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c3a6f4-a77a-44b0-8077-731f9baabc6b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:24.965495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10eb90f-d844-4806-a9b0-c1e5412357f8 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:25.002799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c9955b-0c40-4ee1-b056-978f8ee6bf75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:25.003805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c9955b-0c40-4ee1-b056-978f8ee6bf75 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:25.004334 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c9955b-0c40-4ee1-b056-978f8ee6bf75 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:25.004800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c9955b-0c40-4ee1-b056-978f8ee6bf75 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:25.005838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c9955b-0c40-4ee1-b056-978f8ee6bf75 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:25.006341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c9955b-0c40-4ee1-b056-978f8ee6bf75 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:25.008065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c9955b-0c40-4ee1-b056-978f8ee6bf75 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:25.027370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 288/596] 10.222.0.20 () {56 vars in 1135 bytes} [Wed Oct 19 21:43:24 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:25.029622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb09def-c65c-4c73-a4e7-a8c44b572d8a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:25.030187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb09def-c65c-4c73-a4e7-a8c44b572d8a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:25.030664 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb09def-c65c-4c73-a4e7-a8c44b572d8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:25.056667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-4bb09def-c65c-4c73-a4e7-a8c44b572d8a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:25.058435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4bb09def-c65c-4c73-a4e7-a8c44b572d8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:25.067490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c3a6f4-a77a-44b0-8077-731f9baabc6b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:25.114861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:25.116061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:25.116937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:25.117644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:25.118553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] RBAC: Adding query filter params (interface=public, service_id=4987247f368a48de9cfc2f07ec2bfa96, region_id=RegionOne) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:25.119302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:25.121253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:25.134377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 309/597] 10.222.0.20 () {56 vars in 1251 bytes} [Wed Oct 19 21:43:24 2022] GET /identity/v3/endpoints?service_id=4987247f368a48de9cfc2f07ec2bfa96&interface=public®ion_id=RegionOne => generated 194 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:25.134377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:25.137785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:25.139773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 289/598] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:25 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:25.140632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb09def-c65c-4c73-a4e7-a8c44b572d8a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:25.195305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:25.294485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb09def-c65c-4c73-a4e7-a8c44b572d8a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:25.327763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b4b2db-2467-4cc2-8629-827e7858823b None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:25.360311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b76e6d9-e045-4b85-82da-b6e70e1e4369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:25.360944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76e6d9-e045-4b85-82da-b6e70e1e4369 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:25.361225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76e6d9-e045-4b85-82da-b6e70e1e4369 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:25.361501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b76e6d9-e045-4b85-82da-b6e70e1e4369 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:25.362124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b76e6d9-e045-4b85-82da-b6e70e1e4369 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:25.368547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b76e6d9-e045-4b85-82da-b6e70e1e4369 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:25.375726 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 290/599] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:43:25 2022] GET /identity/v3/services => generated 772 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:25.382759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eba20c-b0aa-4c75-aa8f-4d5434cf5f65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:25.383459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eba20c-b0aa-4c75-aa8f-4d5434cf5f65 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:25.383738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eba20c-b0aa-4c75-aa8f-4d5434cf5f65 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:25.384004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eba20c-b0aa-4c75-aa8f-4d5434cf5f65 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:25.481820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eba20c-b0aa-4c75-aa8f-4d5434cf5f65 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:25.483314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eba20c-b0aa-4c75-aa8f-4d5434cf5f65 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:25.488043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:25.519756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b76e6d9-e045-4b85-82da-b6e70e1e4369 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:25.632587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 310/600] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:25 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 501 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:43:25.637703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eba20c-b0aa-4c75-aa8f-4d5434cf5f65 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:25.695836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b76e6d9-e045-4b85-82da-b6e70e1e4369 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:25.697686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:25.743565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65858657-00ca-4022-824a-667ef780f35e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:25.744119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65858657-00ca-4022-824a-667ef780f35e None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:25.744391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65858657-00ca-4022-824a-667ef780f35e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:25.744659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65858657-00ca-4022-824a-667ef780f35e None admin] PATH_INFO: `/v3/roles/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:25.756333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65858657-00ca-4022-824a-667ef780f35e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-65858657-00ca-4022-824a-667ef780f35e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:25.762159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:25.763110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:25.764149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 291/601] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:43:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:25.771907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65858657-00ca-4022-824a-667ef780f35e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:25.772427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eba20c-b0aa-4c75-aa8f-4d5434cf5f65 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:25.775623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:25.830930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:25.831964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:25.832502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:25.833132 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:25.855544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:25.857611 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:25.885087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:25.886627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:25.887555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:25.887555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:25.887555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:25.893180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 311/602] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:25.897250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:25.973201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65858657-00ca-4022-824a-667ef780f35e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.024795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b4750-5208-452e-8efa-588287406526 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.052349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.052878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.053161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.053438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.054124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] RBAC: Adding query filter params (name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:26.054429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:26.055979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:26.064294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.065205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.065677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.066178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.067043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:26.067556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:26.069123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:26.075457 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 292/603] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:43:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:26.089381 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:26.091936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 312/604] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:26.100086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:26.127954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:26.129299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:26.270145 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ef9711-5f4e-4c7d-be53-2c3610f35fc7 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.275012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3cdb101-0778-4626-aeac-bc5f4beb8e2b None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.311371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.312103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.312391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.312666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.315702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e96f0d-ec78-4de0-9861-1acb77181ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.316491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e96f0d-ec78-4de0-9861-1acb77181ed8 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.316912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e96f0d-ec78-4de0-9861-1acb77181ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.317327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e96f0d-ec78-4de0-9861-1acb77181ed8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.323902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:26.325308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e96f0d-ec78-4de0-9861-1acb77181ed8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:26.325427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:26.327250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e96f0d-ec78-4de0-9861-1acb77181ed8 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-69e96f0d-ec78-4de0-9861-1acb77181ed8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:26.341459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:26.342600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:26.343715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:26.343715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:26.343715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:26.343715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:26.343715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:26.343715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:26.343715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:26.343715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:26.344158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:26.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:26.346791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:26.346791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:26.346791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:26.348782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 313/606] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:26.349175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 293/606] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:26.350508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:26.350834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781c4d3-981b-4160-9a29-d7f23de70660 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.351330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781c4d3-981b-4160-9a29-d7f23de70660 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.351783 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781c4d3-981b-4160-9a29-d7f23de70660 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.353648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 314/607] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:26.359657 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c781c4d3-981b-4160-9a29-d7f23de70660 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:26.441010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:26.465749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:26.466929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26d620f-af90-4638-a75d-c6029afef325 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.491150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c781c4d3-981b-4160-9a29-d7f23de70660 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.503156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bfd4ffa-2fa2-4da6-b664-d320f6558f88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.503967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd4ffa-2fa2-4da6-b664-d320f6558f88 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.504450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd4ffa-2fa2-4da6-b664-d320f6558f88 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.505097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bfd4ffa-2fa2-4da6-b664-d320f6558f88 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.505747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfd4ffa-2fa2-4da6-b664-d320f6558f88 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:26.506055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfd4ffa-2fa2-4da6-b664-d320f6558f88 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:26.507570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bfd4ffa-2fa2-4da6-b664-d320f6558f88 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:26.520955 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 294/608] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:26.525778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b46e2-e05a-4830-847e-3f7cfab0093c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.525778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b46e2-e05a-4830-847e-3f7cfab0093c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.525778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b46e2-e05a-4830-847e-3f7cfab0093c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.531322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e49372-055e-4256-8aa9-3838f697502c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.531992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e49372-055e-4256-8aa9-3838f697502c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.532273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e49372-055e-4256-8aa9-3838f697502c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.532542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e49372-055e-4256-8aa9-3838f697502c None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.533285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e49372-055e-4256-8aa9-3838f697502c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:26.533591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e49372-055e-4256-8aa9-3838f697502c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:26.535034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e49372-055e-4256-8aa9-3838f697502c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:26.543753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-9e7b46e2-e05a-4830-847e-3f7cfab0093c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:26.544932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9e7b46e2-e05a-4830-847e-3f7cfab0093c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:26.549539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 315/609] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:26.550568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80617528-5117-4056-8df3-1cfde2caad2f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.550853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80617528-5117-4056-8df3-1cfde2caad2f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.551153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80617528-5117-4056-8df3-1cfde2caad2f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.552461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 316/610] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:26.553010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80617528-5117-4056-8df3-1cfde2caad2f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:26.625351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 295/611] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:26 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:26.626617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7b46e2-e05a-4830-847e-3f7cfab0093c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:26.693143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80617528-5117-4056-8df3-1cfde2caad2f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.706074 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:26.741264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e479240-498e-4e9b-8d1d-eb0a3e50d63a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.741891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e479240-498e-4e9b-8d1d-eb0a3e50d63a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.742170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e479240-498e-4e9b-8d1d-eb0a3e50d63a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.742443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e479240-498e-4e9b-8d1d-eb0a3e50d63a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.743010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7b46e2-e05a-4830-847e-3f7cfab0093c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.746408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:26.751875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e479240-498e-4e9b-8d1d-eb0a3e50d63a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:26.753390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e479240-498e-4e9b-8d1d-eb0a3e50d63a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-0e479240-498e-4e9b-8d1d-eb0a3e50d63a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:26.769883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:26.770854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 21:43:26.771843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:26.775087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 317/612] 10.222.0.20 () {56 vars in 1117 bytes} [Wed Oct 19 21:43:26 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:26.777454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c01d4a2-f642-439e-bbb3-c0821efc6a1b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.779314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c01d4a2-f642-439e-bbb3-c0821efc6a1b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.780517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c01d4a2-f642-439e-bbb3-c0821efc6a1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.802853 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.803491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.803829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.804108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:26.805148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=b19bbdff6ae1428d8cfc7a33d4fafb49) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:26.805460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:26.806897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:26.809125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-9c01d4a2-f642-439e-bbb3-c0821efc6a1b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:26.810577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9c01d4a2-f642-439e-bbb3-c0821efc6a1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:26.813676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 296/613] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:26 2022] GET /identity/v3/role_assignments?user.id=b19bbdff6ae1428d8cfc7a33d4fafb49&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 608 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:26.823548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:26.945501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b972216-e61d-4282-996a-dce0b1edd742 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:26.962644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 318/614] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:26 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:26.963374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c01d4a2-f642-439e-bbb3-c0821efc6a1b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:26.981058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:26.997722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:26.998720 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:26.999229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:26.999710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.000805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:27.012052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:27.012052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:27.041266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 297/615] 10.222.0.20 () {56 vars in 1149 bytes} [Wed Oct 19 21:43:26 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:27.048040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:27.080616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c01d4a2-f642-439e-bbb3-c0821efc6a1b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:27.112150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:27.122221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:27.122852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:27.123728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:27.124001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.124735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:27.126371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:27.143201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:27.144180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:27.145058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 319/616] 10.222.0.20 () {56 vars in 1132 bytes} [Wed Oct 19 21:43:26 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:27.152174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:27.187277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b169e248-053d-4312-90ec-f6e647f99c5a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:27.233026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baa969f-5a89-4bf8-977d-189ddabbd569 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:27.233829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baa969f-5a89-4bf8-977d-189ddabbd569 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:27.234104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baa969f-5a89-4bf8-977d-189ddabbd569 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:27.234376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7baa969f-5a89-4bf8-977d-189ddabbd569 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.334360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8f9d34-d112-486f-b8d3-45cbb8b72a90 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:27.334994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:27.348360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baa969f-5a89-4bf8-977d-189ddabbd569 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:27.355756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7baa969f-5a89-4bf8-977d-189ddabbd569 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:27.372378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:27.373276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:27.373737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:27.374187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.375054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:27.375569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:27.377205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:27.383438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 320/617] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:27 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:27.384537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:27.444820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:27.462710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 298/618] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:27 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:43:27.463743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baa969f-5a89-4bf8-977d-189ddabbd569 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:27.497651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fecf360-13bb-451b-9a91-7255dc3de9a0 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:27.535153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:27.536179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:27.536642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:27.537096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.537856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:27.539706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:27.549426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:27.550435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 19 21:43:27.551507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:27.558809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:27.559053 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 321/619] 10.222.0.20 () {56 vars in 1129 bytes} [Wed Oct 19 21:43:27 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:27.560744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:27.569818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7baa969f-5a89-4bf8-977d-189ddabbd569 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:27.613946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:27.614793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:27.615329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:27.615754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.617091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:27.617847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:27.620197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:27.626981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 299/620] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:27 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:27.628168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:27.722826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab88094f-36c0-47a5-b1aa-5188e1bd1e4c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:27.748194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:27.768388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:27.769372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:27.769813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:27.770232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.784220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:27.786111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:27.814472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:27.815697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:27.816589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:27.816589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:27.816589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:27.816589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 322/621] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:27.823931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:27.847466 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0b8147-13b5-45ce-9783-95c75be91aee None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:27.900253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ee4ff2-9a21-4d17-82d7-9bfb0f1483fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:27.901708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ee4ff2-9a21-4d17-82d7-9bfb0f1483fd None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:27.901994 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ee4ff2-9a21-4d17-82d7-9bfb0f1483fd None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:27.902861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ee4ff2-9a21-4d17-82d7-9bfb0f1483fd None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.903851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ee4ff2-9a21-4d17-82d7-9bfb0f1483fd None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:27.904166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ee4ff2-9a21-4d17-82d7-9bfb0f1483fd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:27.905789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ee4ff2-9a21-4d17-82d7-9bfb0f1483fd None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:27.916129 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 300/622] 10.222.0.20 () {56 vars in 1124 bytes} [Wed Oct 19 21:43:27 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:27.918225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c23c3-420c-4c1d-b6f4-4b54dd7d7df8 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:27.918508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c23c3-420c-4c1d-b6f4-4b54dd7d7df8 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:27.918779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8c23c3-420c-4c1d-b6f4-4b54dd7d7df8 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:27.920182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 301/623] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:27.920939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8c23c3-420c-4c1d-b6f4-4b54dd7d7df8 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:27.948920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:27.985034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de13d70e-5a7c-423b-be3c-b1c7b4202be9 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:28.030284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f19955-3647-412a-a0c5-cdc55e15e641 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:28.030943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f19955-3647-412a-a0c5-cdc55e15e641 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.031260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f19955-3647-412a-a0c5-cdc55e15e641 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.031531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f19955-3647-412a-a0c5-cdc55e15e641 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.032225 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f19955-3647-412a-a0c5-cdc55e15e641 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:28.033073 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f19955-3647-412a-a0c5-cdc55e15e641 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:28.034783 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f19955-3647-412a-a0c5-cdc55e15e641 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:28.047619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8c23c3-420c-4c1d-b6f4-4b54dd7d7df8 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:28.049607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 323/624] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:28.052258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06357a7-ad1c-4091-8d22-bfe70f339684 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.052544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06357a7-ad1c-4091-8d22-bfe70f339684 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.052815 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06357a7-ad1c-4091-8d22-bfe70f339684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.079011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a06357a7-ad1c-4091-8d22-bfe70f339684 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:28.080200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a06357a7-ad1c-4091-8d22-bfe70f339684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:28.094904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:28.095595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.095980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.096252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.096999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:28.097343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:28.098698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:28.104411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 302/625] 10.222.0.20 () {56 vars in 1124 bytes} [Wed Oct 19 21:43:27 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:28.105323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:28.107173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:28.156890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 324/626] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:28 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:28.157748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06357a7-ad1c-4091-8d22-bfe70f339684 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:28.235678 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:28.264188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4e752e-bd8f-4915-915d-f6b5fa9c80b0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:28.278826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06357a7-ad1c-4091-8d22-bfe70f339684 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:28.327459 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:28.328315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.328586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.328854 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.340266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:28.343140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:28.346474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92c0ea-9a7e-49db-9aa3-7548d8e56b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:28.347102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92c0ea-9a7e-49db-9aa3-7548d8e56b4e None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.347377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92c0ea-9a7e-49db-9aa3-7548d8e56b4e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.347644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b92c0ea-9a7e-49db-9aa3-7548d8e56b4e None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.352422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b92c0ea-9a7e-49db-9aa3-7548d8e56b4e None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:28.352422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b92c0ea-9a7e-49db-9aa3-7548d8e56b4e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:28.366931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:28.369167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:28.371398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 303/627] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:28.374874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:28.467122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 325/628] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:43:28 2022] POST /identity/v3/endpoints => generated 346 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:28.489697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68074bd8-76ac-454c-b266-23d795d1ca34 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:28.509576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92c0ea-9a7e-49db-9aa3-7548d8e56b4e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:28.528591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:28.532963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:28.533777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.533911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.534088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.534772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:28.535104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:28.536752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:28.545059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:28.550077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 304/629] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:28.558486 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:28.630661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b92c0ea-9a7e-49db-9aa3-7548d8e56b4e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:28.665405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa575e5d-f2fd-4f9d-a0bd-43ffdf63b8e2 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:28.670908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:28.671876 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.672356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.672840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] PATH_INFO: `/v3/services/network` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.673628 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:28.675224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:28.687055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:28.688181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 19 21:43:28.689015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:28.691978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 326/630] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:28 2022] GET /identity/v3/services/network => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:28.698850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:28.706162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:28.706768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.707152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.707422 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.708154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] RBAC: Adding query filter params (role.id=fa322a306cb442c79d45a314b16f1b85, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=eba069a7652d4c26a6a35a73cb2be767) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:28.708460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:28.709952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:28.716585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 305/631] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:28 2022] GET /identity/v3/role_assignments?user.id=eba069a7652d4c26a6a35a73cb2be767&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=fa322a306cb442c79d45a314b16f1b85 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:28.809917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:28.832448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5801fcc-b31a-475a-a10c-c1262b461c19 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:28.870894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:28.871887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:28.872368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:28.872839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:28.873726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] RBAC: Adding query filter params (name=network) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:28.874228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:28.875885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:28.881835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 327/632] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:28 2022] GET /identity/v3/services?name=network => generated 124 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:28.888349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:28.954710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:28.999860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a138bb04-93f8-40ca-b836-abc11fcf0efe None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:29.036154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70153a2d-afaa-46fd-b2c6-5a981ab143bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:29.036821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70153a2d-afaa-46fd-b2c6-5a981ab143bc None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:29.037046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70153a2d-afaa-46fd-b2c6-5a981ab143bc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:29.037319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70153a2d-afaa-46fd-b2c6-5a981ab143bc None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:29.037960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70153a2d-afaa-46fd-b2c6-5a981ab143bc None admin] RBAC: Adding query filter params (type=network) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:29.038262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70153a2d-afaa-46fd-b2c6-5a981ab143bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:29.039673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70153a2d-afaa-46fd-b2c6-5a981ab143bc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:29.045835 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 306/633] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:28 2022] GET /identity/v3/services?type=network => generated 350 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:29.051507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:29.180450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec8b847d-ed0a-437a-b53f-6b268a7f3c4c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:29.215760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a6a972-9c55-4f35-b2cc-bb032b345b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:29.216371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a6a972-9c55-4f35-b2cc-bb032b345b87 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:29.216691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a6a972-9c55-4f35-b2cc-bb032b345b87 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:29.216962 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a6a972-9c55-4f35-b2cc-bb032b345b87 None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:29.217636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a6a972-9c55-4f35-b2cc-bb032b345b87 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:29.219182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a6a972-9c55-4f35-b2cc-bb032b345b87 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:29.299359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:29.305653 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 328/634] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:43:29 2022] POST /identity/v3/endpoints => generated 337 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:29.479719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec018b9-74aa-4de1-bd51-3261a8a33633 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:29.480287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec018b9-74aa-4de1-bd51-3261a8a33633 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:29.480287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec018b9-74aa-4de1-bd51-3261a8a33633 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:29.481869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 307/635] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:29.487663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c049c-61c3-49d7-b0e9-381656045169 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:29.487943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c049c-61c3-49d7-b0e9-381656045169 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:29.488276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752c049c-61c3-49d7-b0e9-381656045169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:29.515661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-752c049c-61c3-49d7-b0e9-381656045169 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:29.516941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-752c049c-61c3-49d7-b0e9-381656045169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:29.617599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 329/636] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:29 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:29.908932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec018b9-74aa-4de1-bd51-3261a8a33633 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:30.037828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec018b9-74aa-4de1-bd51-3261a8a33633 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:30.041519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4257fb37-1a22-49cb-a1eb-2e7265f0ca1d None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:30.042362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4257fb37-1a22-49cb-a1eb-2e7265f0ca1d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:30.042657 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4257fb37-1a22-49cb-a1eb-2e7265f0ca1d None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:30.046285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 330/637] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:30 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:30.052120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c15c7e-0983-49e6-ab6a-fe315d3b0217 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:30.052120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c15c7e-0983-49e6-ab6a-fe315d3b0217 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:30.052120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c15c7e-0983-49e6-ab6a-fe315d3b0217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:30.070012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:30.072804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-20c15c7e-0983-49e6-ab6a-fe315d3b0217 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:30.074806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-20c15c7e-0983-49e6-ab6a-fe315d3b0217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:30.081691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:30.082318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:30.088342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:30.088342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:30.088342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:30.088342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:30.096325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:30.097314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:30.098133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:30.100520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 308/638] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:29 2022] GET /identity/v3/services/object-store => generated 93 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:30.106568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:30.153961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 331/639] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:30 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:30.215348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9926eb22-d231-4dfd-8b52-29910f6e9707 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:30.249313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:30.250121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:30.250600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:30.251121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:30.251930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:30.252398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:30.253927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:30.258846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 309/640] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:30 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:30.265329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c15c7e-0983-49e6-ab6a-fe315d3b0217 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:30.324377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:30.373718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c15c7e-0983-49e6-ab6a-fe315d3b0217 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:30.408633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd7961b-577e-47f7-896b-c266d44ee501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:30.409243 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd7961b-577e-47f7-896b-c266d44ee501 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:30.409519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd7961b-577e-47f7-896b-c266d44ee501 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:30.409792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd7961b-577e-47f7-896b-c266d44ee501 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:30.410421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd7961b-577e-47f7-896b-c266d44ee501 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:30.410728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd7961b-577e-47f7-896b-c266d44ee501 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:30.412124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd7961b-577e-47f7-896b-c266d44ee501 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:30.417559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 332/641] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:30 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:30.456483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:30.514872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:30.567607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157e8c72-0024-4e2d-9667-5765001c2f42 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:30.605626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f245b24-2dd7-466d-9c98-c81712fbbadb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:30.606338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f245b24-2dd7-466d-9c98-c81712fbbadb None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:30.606683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f245b24-2dd7-466d-9c98-c81712fbbadb None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:30.607070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f245b24-2dd7-466d-9c98-c81712fbbadb None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:30.607844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f245b24-2dd7-466d-9c98-c81712fbbadb None admin] RBAC: Authorizing `identity:create_service()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:30.609273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f245b24-2dd7-466d-9c98-c81712fbbadb None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:30.681913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 310/642] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:43:30 2022] POST /identity/v3/services => generated 269 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:31.254828 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72fceff-c1ff-495f-9cb4-c6c99b1d309c None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:31.255413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72fceff-c1ff-495f-9cb4-c6c99b1d309c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:31.255697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72fceff-c1ff-495f-9cb4-c6c99b1d309c None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:31.257212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 333/643] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:31.263293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7c9b95-b8f2-4534-9e52-f1f0c80bf193 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:31.263886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7c9b95-b8f2-4534-9e52-f1f0c80bf193 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:31.264393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7c9b95-b8f2-4534-9e52-f1f0c80bf193 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:31.283380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-9e7c9b95-b8f2-4534-9e52-f1f0c80bf193 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:31.284901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9e7c9b95-b8f2-4534-9e52-f1f0c80bf193 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:31.289835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa355ad-c147-45c0-8902-de7388432427 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:31.290321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa355ad-c147-45c0-8902-de7388432427 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:31.290406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa355ad-c147-45c0-8902-de7388432427 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:31.292508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 334/644] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:31.297478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36169eb-a4a0-4327-af01-1958e72182f2 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:31.297753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36169eb-a4a0-4327-af01-1958e72182f2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:31.298020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b36169eb-a4a0-4327-af01-1958e72182f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:31.316516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b36169eb-a4a0-4327-af01-1958e72182f2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:31.317691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b36169eb-a4a0-4327-af01-1958e72182f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:31.367473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 311/645] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:31 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:31.392549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 335/646] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:31 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:31.652189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7c9b95-b8f2-4534-9e52-f1f0c80bf193 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:31.696193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:31.710254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36169eb-a4a0-4327-af01-1958e72182f2 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:31.803306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:31.816982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7c9b95-b8f2-4534-9e52-f1f0c80bf193 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:31.881539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c4c7b-a690-4a75-a0b1-aa84b97d0105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:31.882167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c4c7b-a690-4a75-a0b1-aa84b97d0105 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:31.882442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c4c7b-a690-4a75-a0b1-aa84b97d0105 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:31.882713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-294c4c7b-a690-4a75-a0b1-aa84b97d0105 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:31.891642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b36169eb-a4a0-4327-af01-1958e72182f2 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:31.901137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-294c4c7b-a690-4a75-a0b1-aa84b97d0105 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-294c4c7b-a690-4a75-a0b1-aa84b97d0105 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:31.907846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:31.908829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 21:43:31.909790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:31.913927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 312/647] 10.222.0.20 () {56 vars in 1120 bytes} [Wed Oct 19 21:43:31 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:31.918005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c4c7b-a690-4a75-a0b1-aa84b97d0105 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:31.938412 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:31.939437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:31.940089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:31.940569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:31.941353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:31.943088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:31.954358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:31.955442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:31.956267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:31.964931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 336/648] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:31 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:31.966469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:32.076339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96a91f8-448d-41b1-b137-5e9b2bc54649 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:32.104115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:32.130607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:32.131441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.131815 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.132184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.132915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:32.133371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:32.134883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:32.140660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 337/649] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:31 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:32.148913 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:32.165503 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:32.169871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-294c4c7b-a690-4a75-a0b1-aa84b97d0105 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:32.209236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:32.209943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.210281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.210615 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.211468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:32.211907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:32.213497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:32.227022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 313/650] 10.222.0.20 () {56 vars in 1115 bytes} [Wed Oct 19 21:43:31 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:32.228718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:32.282475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47b73757-041c-4392-9c1c-1c1db95a3862 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:32.322574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:32.323362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.323738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.324123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.324853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:32.325260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:32.327254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:32.334984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 338/651] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:32 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:32.350360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568096a9-6ec0-40d6-a743-9fd0c2d8f16f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:32.389814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:32.390508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.390784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.391080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.404553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:32.406039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:32.422520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:32.423564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:32.424688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:32.424688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:32.424688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:32.424688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 314/652] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:32.431396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:32.481849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:32.545270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e778da10-4af6-4bae-8433-eba05818f7ca None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:32.581491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f27c39-ff72-4528-84cb-33e24cb90088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:32.582103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f27c39-ff72-4528-84cb-33e24cb90088 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.582378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f27c39-ff72-4528-84cb-33e24cb90088 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.582649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f27c39-ff72-4528-84cb-33e24cb90088 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.583313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f27c39-ff72-4528-84cb-33e24cb90088 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:32.583612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f27c39-ff72-4528-84cb-33e24cb90088 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:32.584972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f27c39-ff72-4528-84cb-33e24cb90088 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:32.598104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 339/653] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:32.603833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:32.620736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:32.716330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868de52-e436-4468-9f56-39f2bb0b6f79 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:32.750523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:32.751174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.751457 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.751721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.758929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:32.760382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:32.768226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e464be0-e905-4fea-8f05-a51cd232e744 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.768515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e464be0-e905-4fea-8f05-a51cd232e744 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.768791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e464be0-e905-4fea-8f05-a51cd232e744 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.770360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 340/654] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:32.772546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:32.773483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:32.775905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 315/655] 10.222.0.20 () {56 vars in 1117 bytes} [Wed Oct 19 21:43:32 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:32.778143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631f905-cbf0-4d8c-a9af-d809c9b4c3fb None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.778143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631f905-cbf0-4d8c-a9af-d809c9b4c3fb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.778143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5631f905-cbf0-4d8c-a9af-d809c9b4c3fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.784357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:32.804291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5631f905-cbf0-4d8c-a9af-d809c9b4c3fb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:32.805858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5631f905-cbf0-4d8c-a9af-d809c9b4c3fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:32.871626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:32.887097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 341/656] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:32 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:32.911797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70baf32b-b3c7-4f89-9d33-cc6e43e85180 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:32.934877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5149838b-3d57-49ae-b276-a1b892d52389 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.935188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5149838b-3d57-49ae-b276-a1b892d52389 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.935460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5149838b-3d57-49ae-b276-a1b892d52389 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.937433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 342/657] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:32.942159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3c751-ea43-4aaf-b98c-b6142d2c92c0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.942441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3c751-ea43-4aaf-b98c-b6142d2c92c0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.942724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5a3c751-ea43-4aaf-b98c-b6142d2c92c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.951654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:32.952260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:32.952539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:32.952812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:32.953554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:32.953860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:32.955315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:32.968580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f5a3c751-ea43-4aaf-b98c-b6142d2c92c0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:32.970528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f5a3c751-ea43-4aaf-b98c-b6142d2c92c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:32.973589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 316/658] 10.222.0.20 () {56 vars in 1149 bytes} [Wed Oct 19 21:43:32 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:32.983918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.001290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:33.054369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 343/659] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:32 2022] POST /identity/v3/auth/tokens => generated 1271 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:33.093812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2bc1cd-0119-4622-928f-1bb9542b8b34 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.124590 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:33.130661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.131324 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.131606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.138049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.142507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.144084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:33.159745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:33.160673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:33.161539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:33.161539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:33.161539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:33.163065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 317/660] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:33.166761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.167461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3c751-ea43-4aaf-b98c-b6142d2c92c0 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.291177 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5a3c751-ea43-4aaf-b98c-b6142d2c92c0 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.300877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465edff2-6805-45c4-bc16-45b4f345cdf7 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.329318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba348261-8bb7-4bf4-b51e-a1d9a1523497 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.330002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba348261-8bb7-4bf4-b51e-a1d9a1523497 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.330198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba348261-8bb7-4bf4-b51e-a1d9a1523497 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.330470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba348261-8bb7-4bf4-b51e-a1d9a1523497 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.331228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba348261-8bb7-4bf4-b51e-a1d9a1523497 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.332507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba348261-8bb7-4bf4-b51e-a1d9a1523497 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:33.344400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.345002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.345284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.345556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.346202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:33.346506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.347919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:33.359402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 318/661] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:33.360177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.402034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:33.414146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 344/662] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:43:33 2022] POST /identity/v3/services => generated 241 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:33.414146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba348261-8bb7-4bf4-b51e-a1d9a1523497 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.447029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:33.489117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ca98fd-57b7-478f-b436-7ae40f36904a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.543738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.544699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.545258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.545539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.546149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.547609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:33.547715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba348261-8bb7-4bf4-b51e-a1d9a1523497 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:33.565064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:33.566082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:33.566968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 319/663] 10.222.0.20 () {56 vars in 1132 bytes} [Wed Oct 19 21:43:33 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:33.572795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.594608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.595365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.595962 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.596243 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.605581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.607583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:33.620883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:33.621803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:33.622686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 345/664] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:33.627849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.702056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a3bc10-a3fd-4cbf-ae4c-f2293c9b60ad None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.704270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:33.723705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:33.742640 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c681d2-ec3e-43fe-b2bd-172408a9f768 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.750613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.751928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.752752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.753245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.754554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:33.755156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.757292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:33.768581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 320/665] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:33 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:33.770353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.796664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.797702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.798190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.798785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.799720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:33.800380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.802725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:33.823607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 346/666] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:33.826165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.894475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1e4576-b563-4959-b88f-11f16e049512 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.929880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.930680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.931022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.931410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.932147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:33.933096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.934669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:33.940524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 321/667] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:33 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:33.943377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcf1f39-93b1-4e6e-ac8b-a0296d323458 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:33.945788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:33.977565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:33.990806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e437837-ce04-4612-bc05-c3d3f72a592f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:33.992237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e437837-ce04-4612-bc05-c3d3f72a592f None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:33.992810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e437837-ce04-4612-bc05-c3d3f72a592f None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:33.993278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e437837-ce04-4612-bc05-c3d3f72a592f None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/eba069a7652d4c26a6a35a73cb2be767/roles/85abe52be1cb4e2fa879120f7cebefbe` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:33.996317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e437837-ce04-4612-bc05-c3d3f72a592f None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=eba069a7652d4c26a6a35a73cb2be767, role_id=85abe52be1cb4e2fa879120f7cebefbe)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:33.997748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e437837-ce04-4612-bc05-c3d3f72a592f None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:34.054069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6d5159-7eff-4d6f-8745-e9406f9e0cc0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:34.092097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f5b5d1-47b8-4b9c-a697-7c41cb1cd45a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:34.093001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5b5d1-47b8-4b9c-a697-7c41cb1cd45a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:34.099152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5b5d1-47b8-4b9c-a697-7c41cb1cd45a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:34.099152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f5b5d1-47b8-4b9c-a697-7c41cb1cd45a None admin] PATH_INFO: `/v3/endpoints` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:34.099152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f5b5d1-47b8-4b9c-a697-7c41cb1cd45a None admin] RBAC: Adding query filter params (interface=public, service_id=a9bdf2ac7aec4ff390a3e2ee808bbf00, region_id=RegionOne) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:34.099152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f5b5d1-47b8-4b9c-a697-7c41cb1cd45a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:34.099152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f5b5d1-47b8-4b9c-a697-7c41cb1cd45a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:34.100175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 347/668] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:33 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/eba069a7652d4c26a6a35a73cb2be767/roles/85abe52be1cb4e2fa879120f7cebefbe => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:43:34.105306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 322/669] 10.222.0.20 () {56 vars in 1251 bytes} [Wed Oct 19 21:43:33 2022] GET /identity/v3/endpoints?service_id=a9bdf2ac7aec4ff390a3e2ee808bbf00&interface=public®ion_id=RegionOne => generated 194 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:34.110844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e437837-ce04-4612-bc05-c3d3f72a592f None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:34.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:34.227119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e437837-ce04-4612-bc05-c3d3f72a592f None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:34.269678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ddc85c-2092-434f-b85f-6d5b84602e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:34.270298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddc85c-2092-434f-b85f-6d5b84602e80 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:34.270575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddc85c-2092-434f-b85f-6d5b84602e80 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:34.270843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ddc85c-2092-434f-b85f-6d5b84602e80 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:34.271488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ddc85c-2092-434f-b85f-6d5b84602e80 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:34.272886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ddc85c-2092-434f-b85f-6d5b84602e80 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:34.280019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 348/670] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:43:34 2022] GET /identity/v3/services => generated 1258 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 21:43:34.635310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27348e0b-75b7-4c0f-89f2-d3b438cec7b7 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:34.636284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27348e0b-75b7-4c0f-89f2-d3b438cec7b7 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:34.636404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27348e0b-75b7-4c0f-89f2-d3b438cec7b7 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:34.637995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 323/671] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:34.643145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac3e670-2150-4dee-8aa4-db321a2fdb08 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:34.643601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac3e670-2150-4dee-8aa4-db321a2fdb08 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:34.643680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac3e670-2150-4dee-8aa4-db321a2fdb08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:34.662552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-dac3e670-2150-4dee-8aa4-db321a2fdb08 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:34.663781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-dac3e670-2150-4dee-8aa4-db321a2fdb08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:34.725251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:34.733327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 349/672] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:34 2022] POST /identity/v3/auth/tokens => generated 1373 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:35.012823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27348e0b-75b7-4c0f-89f2-d3b438cec7b7 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:35.114179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:35.122659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27348e0b-75b7-4c0f-89f2-d3b438cec7b7 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:35.157762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d5a17e-29e6-43c6-98b4-a37f1cdd6a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:35.158360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d5a17e-29e6-43c6-98b4-a37f1cdd6a61 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:35.158634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d5a17e-29e6-43c6-98b4-a37f1cdd6a61 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:35.158901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d5a17e-29e6-43c6-98b4-a37f1cdd6a61 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:35.159598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d5a17e-29e6-43c6-98b4-a37f1cdd6a61 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:35.160952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d5a17e-29e6-43c6-98b4-a37f1cdd6a61 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:35.242056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 324/673] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:43:35 2022] POST /identity/v3/services => generated 251 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:35.647117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5d09d3-9659-4c19-8c33-1cf9ba04c7f8 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:35.647637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5d09d3-9659-4c19-8c33-1cf9ba04c7f8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:35.647637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5d09d3-9659-4c19-8c33-1cf9ba04c7f8 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:35.649332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 350/674] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:35.655098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1962726-8906-484a-a742-422a65d93db5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:35.655389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1962726-8906-484a-a742-422a65d93db5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:35.655668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1962726-8906-484a-a742-422a65d93db5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:35.675164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f1962726-8906-484a-a742-422a65d93db5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:35.676369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f1962726-8906-484a-a742-422a65d93db5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:35.754026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 325/675] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:35 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:36.042026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5d09d3-9659-4c19-8c33-1cf9ba04c7f8 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:36.150539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5d09d3-9659-4c19-8c33-1cf9ba04c7f8 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:36.189486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1981c9e-412e-43da-9fa1-d1e47deccaab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:36.190118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1981c9e-412e-43da-9fa1-d1e47deccaab None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.190394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1981c9e-412e-43da-9fa1-d1e47deccaab None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.190724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1981c9e-412e-43da-9fa1-d1e47deccaab None admin] PATH_INFO: `/v3/services/object-store` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.191339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1981c9e-412e-43da-9fa1-d1e47deccaab None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:36.192760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1981c9e-412e-43da-9fa1-d1e47deccaab None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-a1981c9e-412e-43da-9fa1-d1e47deccaab None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:36.202570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:36.203583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:36.204397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 351/676] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:36 2022] GET /identity/v3/services/object-store => generated 93 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:36.209922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1962726-8906-484a-a742-422a65d93db5 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:36.301096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:36.321122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1962726-8906-484a-a742-422a65d93db5 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:36.345837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c02880-38ab-41b3-ba01-de03300f7c0d None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.346423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c02880-38ab-41b3-ba01-de03300f7c0d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.346895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c02880-38ab-41b3-ba01-de03300f7c0d None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.348698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 352/677] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:36.354891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ee7cec-8db7-46b2-93a9-9bf89d19e4ac None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.355523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ee7cec-8db7-46b2-93a9-9bf89d19e4ac None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.356088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ee7cec-8db7-46b2-93a9-9bf89d19e4ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.358345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:36.359144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.359430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.359707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.360407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:36.360685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:36.362107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:36.369323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 326/678] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:36 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:36.376288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:36.379767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d9ee7cec-8db7-46b2-93a9-9bf89d19e4ac None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:36.381369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d9ee7cec-8db7-46b2-93a9-9bf89d19e4ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:36.443871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:36.462851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 353/679] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:36 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:36.485716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c304fa94-4478-4e5f-b210-1a3d2d90c921 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:36.488618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b8b2fa-254c-4076-aa90-eba9cf66f1de None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.488911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b8b2fa-254c-4076-aa90-eba9cf66f1de None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.489189 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b8b2fa-254c-4076-aa90-eba9cf66f1de None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.491831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 354/680] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:36.496526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b05d08-f0ce-47ab-a830-9926c2f6a41e None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.496816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b05d08-f0ce-47ab-a830-9926c2f6a41e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.497090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b05d08-f0ce-47ab-a830-9926c2f6a41e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.516151 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-75b05d08-f0ce-47ab-a830-9926c2f6a41e None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:36.517437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-75b05d08-f0ce-47ab-a830-9926c2f6a41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:36.521254 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:36.521862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.522143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.522418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.523090 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:36.523399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:36.524770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:36.530547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 327/681] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:36 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:36.536404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:36.547940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:36.595425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 355/682] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:36 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:36.640366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac355a3-f0e6-4aa0-9256-fc1664c99e06 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:36.675545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:36.676193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.676476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.676749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] PATH_INFO: `/v3/endpoints` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.677453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] RBAC: Adding query filter params (interface=public, service_id=d27c94ba81f64e3fb1bdcbf8e46a2980, region_id=RegionOne) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:36.677763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:36.679290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:36.684902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 328/683] 10.222.0.20 () {56 vars in 1251 bytes} [Wed Oct 19 21:43:36 2022] GET /identity/v3/endpoints?service_id=d27c94ba81f64e3fb1bdcbf8e46a2980&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:36.690619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b05d08-f0ce-47ab-a830-9926c2f6a41e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:36.771188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:36.796164 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75b05d08-f0ce-47ab-a830-9926c2f6a41e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:36.817657 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:36.833043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc6a38-9e7e-42ff-9ae1-994726447166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:36.833649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc6a38-9e7e-42ff-9ae1-994726447166 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.833927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc6a38-9e7e-42ff-9ae1-994726447166 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.834252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc6a38-9e7e-42ff-9ae1-994726447166 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.834854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eebc6a38-9e7e-42ff-9ae1-994726447166 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:36.836413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eebc6a38-9e7e-42ff-9ae1-994726447166 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:36.843004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 356/684] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:43:36 2022] GET /identity/v3/services => generated 1497 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 21:43:36.872482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:36.880884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc6a38-9e7e-42ff-9ae1-994726447166 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:36.917802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f30236-027e-40dd-8089-0eb872ff7188 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:36.969376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ac0a9f-8761-4b3c-a103-bdb78f4191b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:36.970435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ac0a9f-8761-4b3c-a103-bdb78f4191b1 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:36.970716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ac0a9f-8761-4b3c-a103-bdb78f4191b1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:36.971168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ac0a9f-8761-4b3c-a103-bdb78f4191b1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:36.991339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ac0a9f-8761-4b3c-a103-bdb78f4191b1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:36.997123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc6a38-9e7e-42ff-9ae1-994726447166 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-04ac0a9f-8761-4b3c-a103-bdb78f4191b1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:36.999012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:36.999905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 21:43:37.000765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:37.006719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 329/685] 10.222.0.20 () {56 vars in 1120 bytes} [Wed Oct 19 21:43:36 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:37.007724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ac0a9f-8761-4b3c-a103-bdb78f4191b1 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:37.041267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.042480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.042767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.044865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.044865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.046048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:37.056961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:37.058191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:37.063898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 357/686] 10.222.0.20 () {56 vars in 1132 bytes} [Wed Oct 19 21:43:36 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:37.067394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:37.128158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:37.148001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ac0a9f-8761-4b3c-a103-bdb78f4191b1 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:37.182759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:37.187945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d7701f-11ce-4b40-8369-4ec0beafac8f None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:37.189345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.195498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.195498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.195498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.195498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:37.195498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.195498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.204461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 330/687] 10.222.0.20 () {56 vars in 1115 bytes} [Wed Oct 19 21:43:37 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:37.210130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:37.232440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.233317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.233746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.234158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.234925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:37.235945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.237868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.244355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 358/688] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:37 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:37.249801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:37.340892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e6dc98-0e2a-4762-82aa-d398a10be1e4 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:37.372678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12e9377-ba6b-4533-9642-af6884a1b591 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:37.388228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:37.397444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.398026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.398303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.398571 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.410560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.412125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.418590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.419530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.420071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.420549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.421564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:37.422111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.423754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:37.427820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:37.428743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:37.429637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:37.429637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:37.429637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:37.430607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 331/689] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:37.431860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 359/690] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:37 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:37.436691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:37.437854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:37.484065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:37.539922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c7d052-e72c-4b0d-bddd-214bb72a8433 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:37.546211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02e0e3e8-c3bb-43ef-9ab5-3d9c69d6e9d5 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:37.577012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4204c0bc-9b5e-4b03-ad00-fc508d112b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.577877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4204c0bc-9b5e-4b03-ad00-fc508d112b95 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.578154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4204c0bc-9b5e-4b03-ad00-fc508d112b95 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.578453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4204c0bc-9b5e-4b03-ad00-fc508d112b95 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.579097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4204c0bc-9b5e-4b03-ad00-fc508d112b95 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:37.579397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4204c0bc-9b5e-4b03-ad00-fc508d112b95 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.580758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4204c0bc-9b5e-4b03-ad00-fc508d112b95 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.581593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3481934-347c-4b1a-9572-3a6cdfa8884b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.582364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3481934-347c-4b1a-9572-3a6cdfa8884b None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.582641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3481934-347c-4b1a-9572-3a6cdfa8884b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.583336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3481934-347c-4b1a-9572-3a6cdfa8884b None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.584026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3481934-347c-4b1a-9572-3a6cdfa8884b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.585367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3481934-347c-4b1a-9572-3a6cdfa8884b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.595104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 332/691] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:37.596369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd91066-c190-4bad-b0b2-afa65db6150d None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.596651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd91066-c190-4bad-b0b2-afa65db6150d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.596921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd91066-c190-4bad-b0b2-afa65db6150d None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.598298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 333/692] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:37.599561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd91066-c190-4bad-b0b2-afa65db6150d None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:37.695061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 360/693] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:43:37 2022] POST /identity/v3/endpoints => generated 347 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:37.695061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273fe81-41bf-49f1-bf9c-953da0d79986 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.695061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273fe81-41bf-49f1-bf9c-953da0d79986 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.695061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273fe81-41bf-49f1-bf9c-953da0d79986 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.715715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b273fe81-41bf-49f1-bf9c-953da0d79986 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:37.716562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd91066-c190-4bad-b0b2-afa65db6150d None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:37.717021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b273fe81-41bf-49f1-bf9c-953da0d79986 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:37.718395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:37.756052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:37.766780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.767700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.767975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.769166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.779212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.781504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:37.806125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:37.818552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 334/694] 10.222.0.20 () {56 vars in 1117 bytes} [Wed Oct 19 21:43:37 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:37.819545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:37.824572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 361/695] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:37 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:37.929094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e304d16d-8f14-45f7-aa28-2ecb449d4272 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:37.965167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:37.965694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:37.965973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:37.966245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:37.967069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:37.967376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:37.968874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:37.987135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 335/696] 10.222.0.20 () {56 vars in 1149 bytes} [Wed Oct 19 21:43:37 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:37.992458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b273fe81-41bf-49f1-bf9c-953da0d79986 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.094064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b273fe81-41bf-49f1-bf9c-953da0d79986 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:38.122798 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.138111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03698b0d-05a7-4f89-a506-637da723faf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:38.138727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03698b0d-05a7-4f89-a506-637da723faf5 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:38.139022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03698b0d-05a7-4f89-a506-637da723faf5 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:38.139298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03698b0d-05a7-4f89-a506-637da723faf5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:38.152594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:38.183311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:38.214065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03698b0d-05a7-4f89-a506-637da723faf5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:38.215722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03698b0d-05a7-4f89-a506-637da723faf5 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:38.254325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d8468d-4d35-4cfd-8159-1057dfa594db None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:38.303946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:38.305170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:38.305900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:38.306419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:38.308137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:38.309758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:38.322185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:38.323802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:38.324626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:38.325354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 362/697] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:37 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:43:38.331033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 336/698] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:38.334125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.334247 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03698b0d-05a7-4f89-a506-637da723faf5 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.397871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:38.440049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03698b0d-05a7-4f89-a506-637da723faf5 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:38.454537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:38.469845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d090a0b9-ff27-4ac3-bfb4-d57668da9bb8 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:38.480319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:38.481292 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:38.481955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:38.482452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:38.483328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:38.483853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:38.486454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:38.497063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 363/699] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:38.498320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.510098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:38.510849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:38.511153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:38.511426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:38.524263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:38.525900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:38.546724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:38.547793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:38.548675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:38.548675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:38.548675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:38.550818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 337/700] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:38.557072 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.617208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3106a9-139a-43b8-a7e1-b403e871c98d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:38.647572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:38.660391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:38.661501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:38.661993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:38.662585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:38.663501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:38.664023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:38.665596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:38.677680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 364/701] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:38.678854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.691175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f28f711d-f9f9-4005-94b9-c27cee674e4b None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:38.727693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:38.728696 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:38.729693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:38.730170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:38.731123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:38.731645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:38.733410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:38.743286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:38.747491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 338/702] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:38.752716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.808531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-503f40c3-874f-4e42-9d73-2906b27c56e8 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:38.848318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:38.848923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:38.849198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:38.849466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:38.850518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] RBAC: Adding query filter params (interface=public, service_id=3df6968059e443d38f5c9de1edbf90c7, region_id=RegionOne) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:38.850827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:38.852538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:38.860104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 365/703] 10.222.0.20 () {56 vars in 1251 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/endpoints?service_id=3df6968059e443d38f5c9de1edbf90c7&interface=public®ion_id=RegionOne => generated 194 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:38.865602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.881403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a325371-9b1a-40e1-8033-39032d1eb341 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:38.926990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:38.927626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:38.927902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:38.928166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:38.931715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:38.938232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:38.939897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:38.954690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:38.959818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:38.960790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:38.960790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:38.960790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:38.960790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:38.960790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:38.960790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:38.960790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:38.960790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:38.961215 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 339/704] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:38.962393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:38.987919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e1be8-81f3-4768-9f06-4cab0b433944 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:39.036007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bc9348-09ba-429b-ba20-e27bc88bd491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:39.036743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bc9348-09ba-429b-ba20-e27bc88bd491 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.037570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bc9348-09ba-429b-ba20-e27bc88bd491 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.037847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bc9348-09ba-429b-ba20-e27bc88bd491 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.038732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bc9348-09ba-429b-ba20-e27bc88bd491 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:39.040934 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:39.041487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bc9348-09ba-429b-ba20-e27bc88bd491 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:39.050153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 366/705] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/services => generated 1497 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 21:43:39.095400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25166f7e-0610-4290-bf03-618c07909dc7 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:39.101830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a7feb-b331-449f-91f6-75253b3366ed None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.102114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a7feb-b331-449f-91f6-75253b3366ed None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.102393 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89a7feb-b331-449f-91f6-75253b3366ed None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.108175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 367/706] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:39.110051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb4ff64-563e-49aa-a638-06e3494f8ef7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.110340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb4ff64-563e-49aa-a638-06e3494f8ef7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.110617 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb4ff64-563e-49aa-a638-06e3494f8ef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.131639 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2fb4ff64-563e-49aa-a638-06e3494f8ef7 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:39.132675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:39.132860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2fb4ff64-563e-49aa-a638-06e3494f8ef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:39.133292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.133569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.133835 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.135009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:39.135331 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:39.137485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:39.150873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 340/707] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:39.158098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:39.225834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 368/708] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:39 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:39.268991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d444880c-0372-4e31-a75b-7fb7cea5e4a0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:39.295338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:39.305658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:39.311318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.311318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.311318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.311318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] RBAC: Adding query filter params (role.id=85abe52be1cb4e2fa879120f7cebefbe, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=eba069a7652d4c26a6a35a73cb2be767) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:39.311318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:39.311318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:39.315819 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 341/709] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:39 2022] GET /identity/v3/role_assignments?user.id=eba069a7652d4c26a6a35a73cb2be767&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=85abe52be1cb4e2fa879120f7cebefbe => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:39.500540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb4ff64-563e-49aa-a638-06e3494f8ef7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:39.554623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:39.607621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb4ff64-563e-49aa-a638-06e3494f8ef7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:39.642937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:39.643594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.643865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.644133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.644714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:39.646066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:39.655928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:39.656908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:39.657911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 369/710] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:39 2022] GET /identity/v3/services/object-store => generated 93 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:39.665554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:39.773845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59cd73a-9a0c-4c03-8613-aa0f45d930f8 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:39.821212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399332af-b408-4dcb-a839-bcdeaf089a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:39.821744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399332af-b408-4dcb-a839-bcdeaf089a65 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.822016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399332af-b408-4dcb-a839-bcdeaf089a65 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.822285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399332af-b408-4dcb-a839-bcdeaf089a65 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.822978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399332af-b408-4dcb-a839-bcdeaf089a65 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:39.823840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399332af-b408-4dcb-a839-bcdeaf089a65 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:39.825234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399332af-b408-4dcb-a839-bcdeaf089a65 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:39.830666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 342/711] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:39 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:39.840054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:39.950302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed3f165-2297-45b8-ab31-bfea7df24f07 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:39.963878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:39.985319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b82bca-bb65-4ddc-b6f4-765723856ae5 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.985600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b82bca-bb65-4ddc-b6f4-765723856ae5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.985873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b82bca-bb65-4ddc-b6f4-765723856ae5 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.988401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 343/712] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:39.989135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:39.989745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.990021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.990333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.990918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:39.991253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:39.992655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:39.992774 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87026d-90ba-4c0c-b256-0fc03ddebdec None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:39.993063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87026d-90ba-4c0c-b256-0fc03ddebdec None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:39.993343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee87026d-90ba-4c0c-b256-0fc03ddebdec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:39.999019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 370/713] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:39 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:40.005701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:40.020598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ee87026d-90ba-4c0c-b256-0fc03ddebdec None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:40.021688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ee87026d-90ba-4c0c-b256-0fc03ddebdec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:40.063308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:40.091554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 344/714] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:39 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:40.124221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb4188f-23f1-4a25-b913-29b514f24746 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:40.164431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c708dc-e431-419b-abe2-2d8328c087ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:40.165371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c708dc-e431-419b-abe2-2d8328c087ed None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:40.165877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c708dc-e431-419b-abe2-2d8328c087ed None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:40.166349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c708dc-e431-419b-abe2-2d8328c087ed None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:40.167837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c708dc-e431-419b-abe2-2d8328c087ed None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:40.169466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c708dc-e431-419b-abe2-2d8328c087ed None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:40.240453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 371/715] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:43:40 2022] POST /identity/v3/endpoints => generated 349 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:40.371511 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee87026d-90ba-4c0c-b256-0fc03ddebdec None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:40.477137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee87026d-90ba-4c0c-b256-0fc03ddebdec None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:40.517221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:40.517848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:40.518123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:40.518389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] PATH_INFO: `/v3/services/compute` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:40.518986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:40.520344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:40.529923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:40.531004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:40.532444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 345/716] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:40 2022] GET /identity/v3/services/compute => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:40.537197 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c708dc-e431-419b-abe2-2d8328c087ed None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:40.569351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:40.639531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c708dc-e431-419b-abe2-2d8328c087ed None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:40.675659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f630ef6-bfd0-4acd-bbbd-07285c80b890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:40.675886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f630ef6-bfd0-4acd-bbbd-07285c80b890 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:40.676116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f630ef6-bfd0-4acd-bbbd-07285c80b890 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:40.677337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f630ef6-bfd0-4acd-bbbd-07285c80b890 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:40.677337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f630ef6-bfd0-4acd-bbbd-07285c80b890 None admin] RBAC: Adding query filter params (name=compute) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:40.677337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f630ef6-bfd0-4acd-bbbd-07285c80b890 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:40.678610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f630ef6-bfd0-4acd-bbbd-07285c80b890 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:40.683929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 372/717] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:40 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:40.689217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:40.724178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:40.797812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6f1dd2-f23c-413c-919c-2d0911eefde3 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:40.833986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11df93f3-1529-4e5a-8804-125bebc914b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:40.834589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11df93f3-1529-4e5a-8804-125bebc914b2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:40.834867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11df93f3-1529-4e5a-8804-125bebc914b2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:40.835164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11df93f3-1529-4e5a-8804-125bebc914b2 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:40.835798 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11df93f3-1529-4e5a-8804-125bebc914b2 None admin] RBAC: Adding query filter params (type=compute) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:40.836101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11df93f3-1529-4e5a-8804-125bebc914b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:40.837476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11df93f3-1529-4e5a-8804-125bebc914b2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:40.842391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 346/718] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:40 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:41.326329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad67906-f421-45ce-9bcd-73ef5fe233ff None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:41.326857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad67906-f421-45ce-9bcd-73ef5fe233ff None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:41.326857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad67906-f421-45ce-9bcd-73ef5fe233ff None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:41.328437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 373/719] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:41.333939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e195a-b553-4e5d-99c4-4ba03debea45 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:41.334229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e195a-b553-4e5d-99c4-4ba03debea45 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:41.334500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e195a-b553-4e5d-99c4-4ba03debea45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:41.355012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-dd6e195a-b553-4e5d-99c4-4ba03debea45 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:41.356195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dd6e195a-b553-4e5d-99c4-4ba03debea45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:41.424015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 347/720] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:41 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:41.663875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab7b92-c7d6-4d9c-9f55-14a43d6f5e93 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:41.664551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab7b92-c7d6-4d9c-9f55-14a43d6f5e93 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:41.665039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ab7b92-c7d6-4d9c-9f55-14a43d6f5e93 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:41.667399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 374/721] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:41.672795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feb90ce-fa0b-45c3-be62-af6bd98aed05 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:41.673306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feb90ce-fa0b-45c3-be62-af6bd98aed05 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:41.673809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0feb90ce-fa0b-45c3-be62-af6bd98aed05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:41.693019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0feb90ce-fa0b-45c3-be62-af6bd98aed05 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:41.694629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0feb90ce-fa0b-45c3-be62-af6bd98aed05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:41.695159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ab7b92-c7d6-4d9c-9f55-14a43d6f5e93 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:41.722209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:41.786259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 348/722] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:41 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:41.825993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ab7b92-c7d6-4d9c-9f55-14a43d6f5e93 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:41.835267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:41.862452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:41.863162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:41.863444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:41.863718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:41.864300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:41.865771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:41.874909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:41.875884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:41.876708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 375/723] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:41 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:41.882263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feb90ce-fa0b-45c3-be62-af6bd98aed05 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:41.985553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0feb90ce-fa0b-45c3-be62-af6bd98aed05 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:42.020900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:42.021499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:42.021775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:42.022038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:42.022708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:42.023037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:42.024405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:42.029752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 349/724] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:41 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:42.035105 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:42.070076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:42.095219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:42.141059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f34d2f0-14d8-4021-968d-a6899a6cc6fe None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:42.186924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:42.187811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:42.188237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:42.188443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0213f9ab-c810-43c4-ab38-5e9a3e6c08f4 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:42.188834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:42.189615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:42.190073 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:42.191708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:42.197733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 376/725] 10.222.0.20 () {56 vars in 1114 bytes} [Wed Oct 19 21:43:42 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:42.203830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:42.230362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:42.231263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:42.231715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:42.232135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] PATH_INFO: `/v3/services/image` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:42.232908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:42.234548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:42.236623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:42.244362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:42.251718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:42.252564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 350/726] 10.222.0.20 () {56 vars in 1105 bytes} [Wed Oct 19 21:43:42 2022] GET /identity/v3/services/image => generated 86 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:42.255924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:42.330123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8e3839-ed56-4a05-b028-310729215c48 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:42.368472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:42.373411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb18846-d13a-4dec-bc20-de0f14e29b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:42.374296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb18846-d13a-4dec-bc20-de0f14e29b3d None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:42.374577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb18846-d13a-4dec-bc20-de0f14e29b3d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:42.374894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb18846-d13a-4dec-bc20-de0f14e29b3d None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:42.375653 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb18846-d13a-4dec-bc20-de0f14e29b3d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:42.377543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb18846-d13a-4dec-bc20-de0f14e29b3d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:42.387412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b62e7ab-2153-4132-9acd-e27f6aebf957 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:42.424407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:42.425780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:42.426328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:42.426796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:42.427697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] RBAC: Adding query filter params (name=image) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:42.428182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:42.429701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:42.439397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 351/727] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:42 2022] GET /identity/v3/services?name=image => generated 122 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:42.440705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:42.486416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 377/728] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:43:42 2022] POST /identity/v3/endpoints => generated 346 bytes in 278 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:43:42.563008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bd14b-82cd-4e3f-8135-16da6f1e88f5 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:42.563299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bd14b-82cd-4e3f-8135-16da6f1e88f5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:42.563506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901bd14b-82cd-4e3f-8135-16da6f1e88f5 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:42.564548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525b6f7-8094-42be-8702-ffe388c25d9d None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:42.565155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 378/729] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:42.570806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6441d938-6ae3-4673-9977-cf1fe49851f7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:42.571147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6441d938-6ae3-4673-9977-cf1fe49851f7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:42.571410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6441d938-6ae3-4673-9977-cf1fe49851f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:42.591541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6441d938-6ae3-4673-9977-cf1fe49851f7 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:42.592756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6441d938-6ae3-4673-9977-cf1fe49851f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:42.624495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:42.625705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:42.626209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:42.626942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:42.628562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] RBAC: Adding query filter params (type=image) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:42.629202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:42.630759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:42.637910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 352/730] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:42 2022] GET /identity/v3/services?type=image => generated 122 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:42.685057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 379/731] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:42 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:42.962288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:42.975443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:43.075151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ceb554-5534-47fc-9dfe-e57b4ddcc002 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:43.109832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a0e94-c72f-47c9-bf92-e765d0d7961f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:43.110206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a0e94-c72f-47c9-bf92-e765d0d7961f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:43.110582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899a0e94-c72f-47c9-bf92-e765d0d7961f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:43.112382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:43.112516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 380/732] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:43.113011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:43.113293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:43.113565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:43.114161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:43.115576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:43.118086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3cd27b-1bb4-45ff-b9da-ba00de365150 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:43.118714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3cd27b-1bb4-45ff-b9da-ba00de365150 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:43.119019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3cd27b-1bb4-45ff-b9da-ba00de365150 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:43.125428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:43.126402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:43.127261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 353/733] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:42 2022] GET /identity/v3/services/object-store => generated 93 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:43.133111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:43.144605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3f3cd27b-1bb4-45ff-b9da-ba00de365150 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:43.146191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3f3cd27b-1bb4-45ff-b9da-ba00de365150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:43.149690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:43.222389 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 381/734] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:43 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:43.224050 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:43.251616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa311d2b-8051-4b02-bb27-a5b94e7d08b4 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:43.287476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:43.288133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:43.288460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:43.288731 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:43.289427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:43.289728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:43.291167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:43.296659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 354/735] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:43 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:43.302387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3cd27b-1bb4-45ff-b9da-ba00de365150 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:43.407834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3cd27b-1bb4-45ff-b9da-ba00de365150 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:43.442691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:43.443387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:43.443662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:43.443933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:43.444563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:43.444860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:43.446226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:43.451845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 382/736] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:43 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:43.457613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:43.496228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:43.500449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:43.565627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7557d53-00bb-4586-bccc-20b58068ef26 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:43.600186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:43.600812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:43.601185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:43.601713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:43.602428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] RBAC: Adding query filter params (interface=admin, service_id=d27c94ba81f64e3fb1bdcbf8e46a2980, region_id=RegionOne) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:43.602734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:43.604168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:43.610014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 355/737] 10.222.0.20 () {56 vars in 1249 bytes} [Wed Oct 19 21:43:43 2022] GET /identity/v3/endpoints?service_id=d27c94ba81f64e3fb1bdcbf8e46a2980&interface=admin®ion_id=RegionOne => generated 193 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:43.616535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:43.619734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:43.631253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2910fcd-f439-494d-b5e1-09c88bbbaa67 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:43.670987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73117f51-dbd6-4ee8-afa3-491d83eef4ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:43.671856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73117f51-dbd6-4ee8-afa3-491d83eef4ed None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:43.672415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73117f51-dbd6-4ee8-afa3-491d83eef4ed None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:43.672869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73117f51-dbd6-4ee8-afa3-491d83eef4ed None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:43.673774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73117f51-dbd6-4ee8-afa3-491d83eef4ed None admin] RBAC: Authorizing `identity:create_service()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:43.675340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73117f51-dbd6-4ee8-afa3-491d83eef4ed None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:43.728301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b625ff-9bdb-41ee-b9d1-491b430299a0 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:43.754823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 383/738] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:43:43 2022] POST /identity/v3/services => generated 242 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:43:43.771306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4baaaf-1d10-462b-b549-035ef907c347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:43.771952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4baaaf-1d10-462b-b549-035ef907c347 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:43.772206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4baaaf-1d10-462b-b549-035ef907c347 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:43.772473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4baaaf-1d10-462b-b549-035ef907c347 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:43.773097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4baaaf-1d10-462b-b549-035ef907c347 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:43.774560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df4baaaf-1d10-462b-b549-035ef907c347 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:43.794403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 356/739] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:43:43 2022] GET /identity/v3/services => generated 1727 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 21:43:44.824649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bb66cc-6c0f-4102-b6f3-6787bc1d7ccf None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:44.825156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bb66cc-6c0f-4102-b6f3-6787bc1d7ccf None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:44.825232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bb66cc-6c0f-4102-b6f3-6787bc1d7ccf None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:44.826771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 384/740] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:44.831913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e5e494-b7a4-42cb-8fa1-ff85bf6fd563 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:44.832300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e5e494-b7a4-42cb-8fa1-ff85bf6fd563 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:44.832490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e5e494-b7a4-42cb-8fa1-ff85bf6fd563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:44.839792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d513c4cb-62d4-4299-bb2e-fdcb4be2010c None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:44.840077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d513c4cb-62d4-4299-bb2e-fdcb4be2010c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:44.840355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d513c4cb-62d4-4299-bb2e-fdcb4be2010c None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:44.841837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 385/741] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:44.847853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03327dcd-91dc-42bb-8d18-00840acda719 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:44.848253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03327dcd-91dc-42bb-8d18-00840acda719 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:44.848634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03327dcd-91dc-42bb-8d18-00840acda719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:44.852511 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-41e5e494-b7a4-42cb-8fa1-ff85bf6fd563 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:44.853839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-41e5e494-b7a4-42cb-8fa1-ff85bf6fd563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:44.867668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-03327dcd-91dc-42bb-8d18-00840acda719 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:44.868951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-03327dcd-91dc-42bb-8d18-00840acda719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:44.920728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:44.940687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:44.940878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 357/742] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:44 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:44.952702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 386/743] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:44 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:45.219260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e5e494-b7a4-42cb-8fa1-ff85bf6fd563 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:45.221337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03327dcd-91dc-42bb-8d18-00840acda719 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:45.332690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03327dcd-91dc-42bb-8d18-00840acda719 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:45.341656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e5e494-b7a4-42cb-8fa1-ff85bf6fd563 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:45.379549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3855f9d9-7f72-4470-a2d5-ea91d89f0348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:45.380157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855f9d9-7f72-4470-a2d5-ea91d89f0348 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:45.380430 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855f9d9-7f72-4470-a2d5-ea91d89f0348 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:45.380701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855f9d9-7f72-4470-a2d5-ea91d89f0348 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:45.381800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855f9d9-7f72-4470-a2d5-ea91d89f0348 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:45.383531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855f9d9-7f72-4470-a2d5-ea91d89f0348 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:45.386839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:45.388053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:45.389436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:45.390046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:45.390976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:45.392624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:45.395309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING keystone.server.flask.common [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 19 21:43:45.452292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.sql.core [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 21:43:45.452292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 21:43:45.452292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [parameters: {'id': '68314045098c4c4db9bbc15188cd55bd', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 19 21:43:45.452292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=83022) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 68314045098c4c4db9bbc15188cd55bd.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 68314045098c4c4db9bbc15188cd55bd. Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:43:45.470512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application result.read() Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 21:43:45.471566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application yield resource Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise self.value Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application session.commit() Oct 19 21:43:45.472512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.session.flush() Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application compat.raise_( Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 21:43:45.473386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 21:43:45.474302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application result.read() Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:43:45.475121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [parameters: {'id': '68314045098c4c4db9bbc15188cd55bd', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:45.476069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:45.476954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 68314045098c4c4db9bbc15188cd55bd. Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 358/744] 10.222.0.20 () {60 vars in 1140 bytes} [Wed Oct 19 21:43:45 2022] POST /identity/v3/projects => generated 262 bytes in 253 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:45.480562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:45.495505 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 387/745] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:43:45 2022] POST /identity/v3/services => generated 242 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:43:45.542578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:45.606071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4810967-46b8-4f36-ba3c-041be52c4b3e None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:45.659995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d3a028-b988-4eac-bf71-39670f40fdc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:45.661003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3a028-b988-4eac-bf71-39670f40fdc8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:45.661498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3a028-b988-4eac-bf71-39670f40fdc8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:45.662020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d3a028-b988-4eac-bf71-39670f40fdc8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:45.670692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d3a028-b988-4eac-bf71-39670f40fdc8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:45.672439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d3a028-b988-4eac-bf71-39670f40fdc8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-00d3a028-b988-4eac-bf71-39670f40fdc8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:45.684359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:45.685394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:45.686458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:45.686458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:45.686458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:45.686458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:45.686458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:45.686458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:45.686458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:45.686458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:45.688177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 359/746] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:45.694093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3855f9d9-7f72-4470-a2d5-ea91d89f0348 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:45.756228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:45.813842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3855f9d9-7f72-4470-a2d5-ea91d89f0348 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:45.850231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f6c095-11fe-4dfb-84a1-0dadc6b00a24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:45.851018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f6c095-11fe-4dfb-84a1-0dadc6b00a24 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:45.851202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f6c095-11fe-4dfb-84a1-0dadc6b00a24 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:45.851403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f6c095-11fe-4dfb-84a1-0dadc6b00a24 None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:45.852075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f6c095-11fe-4dfb-84a1-0dadc6b00a24 None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:45.852375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f6c095-11fe-4dfb-84a1-0dadc6b00a24 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:45.853797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f6c095-11fe-4dfb-84a1-0dadc6b00a24 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:45.872137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 388/747] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:45 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:46.036271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca777c1-0d44-40d2-9d22-cb7c31180659 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.036972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca777c1-0d44-40d2-9d22-cb7c31180659 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.036972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca777c1-0d44-40d2-9d22-cb7c31180659 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.038369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 360/748] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:46.044972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d024c0b-65f8-4631-a041-f1419b506157 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.045270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d024c0b-65f8-4631-a041-f1419b506157 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.045529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d024c0b-65f8-4631-a041-f1419b506157 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.051051 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a6622-14fa-4d70-af06-2ba873c663ba None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.051338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a6622-14fa-4d70-af06-2ba873c663ba None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.051663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091a6622-14fa-4d70-af06-2ba873c663ba None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.054082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 361/749] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:46.059288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14912798-24de-4885-ba64-954bfb21eae3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.059570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14912798-24de-4885-ba64-954bfb21eae3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.059845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14912798-24de-4885-ba64-954bfb21eae3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.065027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-0d024c0b-65f8-4631-a041-f1419b506157 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:46.066229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0d024c0b-65f8-4631-a041-f1419b506157 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:46.079945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-14912798-24de-4885-ba64-954bfb21eae3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:46.081444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-14912798-24de-4885-ba64-954bfb21eae3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:46.148105 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 389/750] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:46 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:46.160089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 362/751] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:46 2022] POST /identity/v3/auth/tokens => generated 1666 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:46.422636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d024c0b-65f8-4631-a041-f1419b506157 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:46.434054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14912798-24de-4885-ba64-954bfb21eae3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:46.473265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:46.483280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:46.546830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d024c0b-65f8-4631-a041-f1419b506157 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:46.554702 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14912798-24de-4885-ba64-954bfb21eae3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:46.586306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:46.587289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.587863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.588478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.589789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:46.591478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:46.597259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:46.597969 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.598245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.598512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] PATH_INFO: `/v3/services/compute` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.599128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:46.600481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:46.602605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:46.603580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 19 21:43:46.604969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:46.607740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 390/752] 10.222.0.20 () {56 vars in 1126 bytes} [Wed Oct 19 21:43:46 2022] GET /identity/v3/services/object-store => generated 93 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:46.609781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:46.610718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:46.611602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 363/753] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:46 2022] GET /identity/v3/services/compute => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:46.617373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:46.618461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:46.727356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86684b7-5988-4915-a040-bf8234652762 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:46.738452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:46.739994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:46.745647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36433a00-f8a6-4b80-805d-0385ed8770e2 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:46.764920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:46.765724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.766123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.766481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.767209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:46.767576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:46.769036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:46.775395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 391/754] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:46 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:46.781609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:46.793553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:46.794011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.794360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.794738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.795571 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] RBAC: Adding query filter params (name=compute) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:46.795946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:46.797336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:46.806991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 364/755] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:46 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:46.808765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:46.921477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c50d29-4160-47a2-9b74-18efc0fdf1f1 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:46.927789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483103d5-d04f-42fd-9b19-ec4d699d0580 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:46.961248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:46.961687 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.961974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.962296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.962978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] RBAC: Adding query filter params (type=compute) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:46.963268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:46.965228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:46.975115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 365/756] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:46 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:46.976414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:46.977278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:46.977940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:46.978171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:46.978442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:46.979714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:46.980031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:46.981400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:46.991036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 392/757] 10.222.0.20 () {56 vars in 1122 bytes} [Wed Oct 19 21:43:46 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:46.996305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:47.017907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:47.039965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:47.114587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe9b8fe-e25d-43ca-bcda-b94542e022a4 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:47.116534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203733bf-00a6-445e-bfe4-708930d547c2 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:47.156672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9fb5d9-8830-48e1-9240-009e2b66d398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:47.157717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9fb5d9-8830-48e1-9240-009e2b66d398 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:47.158229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9fb5d9-8830-48e1-9240-009e2b66d398 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:47.158335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:47.158836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9fb5d9-8830-48e1-9240-009e2b66d398 None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:47.158942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:47.159255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:47.159527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] PATH_INFO: `/v3/endpoints` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:47.160235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] RBAC: Adding query filter params (interface=public, service_id=7ce4fca143674b608d0d56917d5bc5ab, region_id=RegionOne) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:47.160528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9fb5d9-8830-48e1-9240-009e2b66d398 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:47.160628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:47.161893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:47.162462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9fb5d9-8830-48e1-9240-009e2b66d398 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:47.171020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 366/758] 10.222.0.20 () {56 vars in 1251 bytes} [Wed Oct 19 21:43:46 2022] GET /identity/v3/endpoints?service_id=7ce4fca143674b608d0d56917d5bc5ab&interface=public®ion_id=RegionOne => generated 194 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:47.172982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:47.242616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 393/759] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:43:46 2022] POST /identity/v3/endpoints => generated 325 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:47.281193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:47.292632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bc08f1-28f0-47da-86d2-34ea38b3facd None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:47.339360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da5bb88-8edb-4448-bb7f-c516bd7d066f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:47.340013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da5bb88-8edb-4448-bb7f-c516bd7d066f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:47.340287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da5bb88-8edb-4448-bb7f-c516bd7d066f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:47.340557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da5bb88-8edb-4448-bb7f-c516bd7d066f None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:47.341235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da5bb88-8edb-4448-bb7f-c516bd7d066f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:47.342713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da5bb88-8edb-4448-bb7f-c516bd7d066f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:47.350735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 367/760] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:43:47 2022] GET /identity/v3/services => generated 1957 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 21:43:47.776897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bc8095-216e-434f-97cc-a8054e569912 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:47.777285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bc8095-216e-434f-97cc-a8054e569912 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:47.777466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4bc8095-216e-434f-97cc-a8054e569912 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:47.779291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 394/761] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:47.784967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacfa5cf-9d58-4429-8d8f-77d663ba7514 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:47.785122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacfa5cf-9d58-4429-8d8f-77d663ba7514 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:47.785326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aacfa5cf-9d58-4429-8d8f-77d663ba7514 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:47.804150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-aacfa5cf-9d58-4429-8d8f-77d663ba7514 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:47.805341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-aacfa5cf-9d58-4429-8d8f-77d663ba7514 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:47.879380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 368/762] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:47 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:48.151906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bc8095-216e-434f-97cc-a8054e569912 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:48.247348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e28dc-bf2d-420c-b0e4-48ff6b7b87d4 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:48.247886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e28dc-bf2d-420c-b0e4-48ff6b7b87d4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:48.247966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e28dc-bf2d-420c-b0e4-48ff6b7b87d4 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:48.249510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 369/763] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:48.254351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca006186-81d1-4cb8-a51c-9062d72a7bda None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:48.254631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca006186-81d1-4cb8-a51c-9062d72a7bda None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:48.254910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca006186-81d1-4cb8-a51c-9062d72a7bda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:48.255144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:48.262895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4bc8095-216e-434f-97cc-a8054e569912 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:48.273835 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ca006186-81d1-4cb8-a51c-9062d72a7bda None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:48.275171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ca006186-81d1-4cb8-a51c-9062d72a7bda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:48.301010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:48.302110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:48.302561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:48.303033 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] PATH_INFO: `/v3/services/image` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:48.303786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:48.305627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:48.315805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:48.316792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:48.317699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:48.320138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 395/764] 10.222.0.20 () {56 vars in 1105 bytes} [Wed Oct 19 21:43:48 2022] GET /identity/v3/services/image => generated 86 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:48.326285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:48.351056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 370/765] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:48 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:48.433770 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b6b8e3-e8ac-448e-ac5c-56c015e91f7e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:48.467993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:48.468646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:48.468931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:48.469204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:48.469829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] RBAC: Adding query filter params (name=image) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:48.470129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:48.471534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:48.476813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 396/766] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:48 2022] GET /identity/v3/services?name=image => generated 122 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:48.482862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca006186-81d1-4cb8-a51c-9062d72a7bda None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:48.495803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:48.588781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca006186-81d1-4cb8-a51c-9062d72a7bda None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:48.624025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:48.624630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:48.624903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:48.625170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:48.625803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] RBAC: Adding query filter params (type=image) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:48.626099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:48.627523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:48.631995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:48.632704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 371/767] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:48 2022] GET /identity/v3/services?type=image => generated 350 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:48.641041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:48.663418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:48.722352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:48.749270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ec7eac-8b5b-4655-a56e-1882ec25d632 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:48.756691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e973ee-7a5b-45ca-975a-2dceb8f7c2ad None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:48.789244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:48.790093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:48.790570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:48.791103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] PATH_INFO: `/v3/endpoints` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:48.792093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] RBAC: Adding query filter params (interface=public, service_id=eb32cb3e2340480fbe9b9f0db4394bc3, region_id=RegionOne) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:48.792604 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:48.793979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:48.803042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 372/768] 10.222.0.20 () {56 vars in 1251 bytes} [Wed Oct 19 21:43:48 2022] GET /identity/v3/endpoints?service_id=eb32cb3e2340480fbe9b9f0db4394bc3&interface=public®ion_id=RegionOne => generated 194 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:48.805193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:48.813951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:48.814559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:48.814834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:48.815147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:48.815785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:48.817220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:48.819853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING keystone.server.flask.common [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 19 21:43:48.864075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.sql.core [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 21:43:48.864075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 19 21:43:48.864075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [parameters: {'user_id': 'b067120171304bc58871dbaee9fe0a3f', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 19 21:43:48.864075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=83021) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result.read() Oct 19 21:43:48.872098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:43:48.873041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application yield resource Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise self.value Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application session.commit() Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 21:43:48.873959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.session.flush() Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application compat.raise_( Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 21:43:48.874822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:43:48.875733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application result.read() Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:43:48.876539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [parameters: {'user_id': 'b067120171304bc58871dbaee9fe0a3f', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:48.877346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 19 21:43:48.878119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 19 21:43:48.880544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 19 21:43:48.880544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 19 21:43:48.880544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 21:43:48.880544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 21:43:48.880544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 19 21:43:48.880544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:48.880544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 397/769] 10.222.0.20 () {60 vars in 1131 bytes} [Wed Oct 19 21:43:48 2022] POST /identity/v3/users => generated 160 bytes in 242 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:48.880544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:48.976488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:48.992447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ccd6e0-6861-46de-9b2e-9062625e2f6c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:49.029816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:49.030553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:49.030839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:49.031216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:49.038794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:49.040537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:49.053429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:49.054385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 19 21:43:49.055432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:49.063874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 398/770] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:48 2022] GET /identity/v3/users/cinder => generated 84 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:49.064856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:49.065474 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8f407d-2adc-40b1-b6b3-3eeed3731526 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:49.105139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5e5f3b-0486-47fa-b34c-c39b1b5cd1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:49.106120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5e5f3b-0486-47fa-b34c-c39b1b5cd1f3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:49.106601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5e5f3b-0486-47fa-b34c-c39b1b5cd1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:49.107105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5e5f3b-0486-47fa-b34c-c39b1b5cd1f3 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:49.108058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5e5f3b-0486-47fa-b34c-c39b1b5cd1f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:49.109682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5e5f3b-0486-47fa-b34c-c39b1b5cd1f3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:49.123055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 373/771] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:43:48 2022] GET /identity/v3/services => generated 1957 bytes in 313 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 21:43:49.175717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18568a6-0200-478e-a878-bf8141129c23 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:49.213763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d334d528-f7b8-4c2d-8092-3ac38b02575e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:49.214382 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d334d528-f7b8-4c2d-8092-3ac38b02575e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:49.214657 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d334d528-f7b8-4c2d-8092-3ac38b02575e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:49.214927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d334d528-f7b8-4c2d-8092-3ac38b02575e None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:49.215662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d334d528-f7b8-4c2d-8092-3ac38b02575e None admin] RBAC: Adding query filter params (name=cinder) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:49.215963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d334d528-f7b8-4c2d-8092-3ac38b02575e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:49.217445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d334d528-f7b8-4c2d-8092-3ac38b02575e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:49.260164 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 399/772] 10.222.0.20 () {56 vars in 1101 bytes} [Wed Oct 19 21:43:49 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:49.516895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81299b3d-7d13-4355-8d0d-5bcc056c2e8b None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:49.517891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81299b3d-7d13-4355-8d0d-5bcc056c2e8b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:49.518458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81299b3d-7d13-4355-8d0d-5bcc056c2e8b None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:49.519998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 374/773] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:49.525887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d87f7ce-f39a-455f-9db2-088f5be084c1 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:49.526295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d87f7ce-f39a-455f-9db2-088f5be084c1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:49.526478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d87f7ce-f39a-455f-9db2-088f5be084c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:49.534607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:49.549289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6d87f7ce-f39a-455f-9db2-088f5be084c1 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:49.550900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6d87f7ce-f39a-455f-9db2-088f5be084c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:49.622745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 400/774] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:49 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:49.645973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c718cd-6cc3-4350-a0b9-b07dcd595773 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:49.646282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c718cd-6cc3-4350-a0b9-b07dcd595773 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:49.646523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c718cd-6cc3-4350-a0b9-b07dcd595773 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:49.648018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 375/775] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:49.652952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b191c9d8-4e2d-4750-8a37-01a4d4270efb None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:49.653504 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b191c9d8-4e2d-4750-8a37-01a4d4270efb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:49.653779 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b191c9d8-4e2d-4750-8a37-01a4d4270efb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:49.671894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b191c9d8-4e2d-4750-8a37-01a4d4270efb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:49.673083 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b191c9d8-4e2d-4750-8a37-01a4d4270efb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:49.740575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 401/776] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:49 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:49.891777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c718cd-6cc3-4350-a0b9-b07dcd595773 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:49.894640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:49.992800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c718cd-6cc3-4350-a0b9-b07dcd595773 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:50.005886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b191c9d8-4e2d-4750-8a37-01a4d4270efb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:50.028734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6419c5-2aeb-48bf-89dc-bdd949f43958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:50.029662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6419c5-2aeb-48bf-89dc-bdd949f43958 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:50.029941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6419c5-2aeb-48bf-89dc-bdd949f43958 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:50.030213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6419c5-2aeb-48bf-89dc-bdd949f43958 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:50.032041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6419c5-2aeb-48bf-89dc-bdd949f43958 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:50.033691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6419c5-2aeb-48bf-89dc-bdd949f43958 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:50.035419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 376/777] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:49 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:50.041368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6419c5-2aeb-48bf-89dc-bdd949f43958 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:50.077693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:50.115735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:50.115999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b191c9d8-4e2d-4750-8a37-01a4d4270efb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:50.147008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6419c5-2aeb-48bf-89dc-bdd949f43958 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:50.151794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:50.152417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:50.152692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:50.152964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] PATH_INFO: `/v3/services/compute` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:50.153798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:50.155205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:50.163840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:50.164795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 19 21:43:50.165677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:50.169803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 402/778] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:50 2022] GET /identity/v3/services/compute => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:50.170576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:50.184612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db44b961-f18a-4857-ae5e-aff2eadd1654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:50.185212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db44b961-f18a-4857-ae5e-aff2eadd1654 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:50.185489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db44b961-f18a-4857-ae5e-aff2eadd1654 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:50.185758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db44b961-f18a-4857-ae5e-aff2eadd1654 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:50.186407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db44b961-f18a-4857-ae5e-aff2eadd1654 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:50.187862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db44b961-f18a-4857-ae5e-aff2eadd1654 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:50.268870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d74997-de62-461e-ad69-7561c1a9037e None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:50.295053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 377/779] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:43:50 2022] POST /identity/v3/projects => generated 311 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:43:50.296762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:50.305282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:50.305912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:50.306192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:50.306524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:50.307249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] RBAC: Adding query filter params (name=compute) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:50.307544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:50.308971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:50.319742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 403/780] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:50 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:50.321267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db44b961-f18a-4857-ae5e-aff2eadd1654 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:50.434503 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db44b961-f18a-4857-ae5e-aff2eadd1654 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:50.459025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:50.472470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb78efa5-a265-4193-a838-79b715eead1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:50.472808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb78efa5-a265-4193-a838-79b715eead1e None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:50.472984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb78efa5-a265-4193-a838-79b715eead1e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:50.473937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb78efa5-a265-4193-a838-79b715eead1e None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:50.473937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb78efa5-a265-4193-a838-79b715eead1e None admin] RBAC: Adding query filter params (type=compute) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:50.474196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb78efa5-a265-4193-a838-79b715eead1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:50.475709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb78efa5-a265-4193-a838-79b715eead1e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:50.481041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 378/781] 10.222.0.20 () {56 vars in 1112 bytes} [Wed Oct 19 21:43:50 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:50.486561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:50.590383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb1c8f7-e434-4e72-9aa5-eeb3dd3a1b86 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:50.623920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb35518-ad9f-4ece-8a81-652a09421816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:50.624534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb35518-ad9f-4ece-8a81-652a09421816 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:50.624817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb35518-ad9f-4ece-8a81-652a09421816 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:50.625096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb35518-ad9f-4ece-8a81-652a09421816 None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:50.625777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb35518-ad9f-4ece-8a81-652a09421816 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:50.627187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb35518-ad9f-4ece-8a81-652a09421816 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:50.743647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 404/782] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:43:50 2022] POST /identity/v3/endpoints => generated 334 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:51.333321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e403e-c15f-44e2-b88a-1e62f0409f6b None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:51.333844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e403e-c15f-44e2-b88a-1e62f0409f6b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:51.333844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e403e-c15f-44e2-b88a-1e62f0409f6b None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:51.335418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 379/783] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:51.340368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1334ffd3-11f6-4ae0-8b73-65e602e6113a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:51.340751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1334ffd3-11f6-4ae0-8b73-65e602e6113a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:51.340911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1334ffd3-11f6-4ae0-8b73-65e602e6113a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:51.359387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-1334ffd3-11f6-4ae0-8b73-65e602e6113a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:51.360585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1334ffd3-11f6-4ae0-8b73-65e602e6113a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:51.409310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:51.439001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 405/784] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:51 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:51.705526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e403e-c15f-44e2-b88a-1e62f0409f6b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:51.808945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e403e-c15f-44e2-b88a-1e62f0409f6b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:51.843102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:51.843722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:51.843998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:51.844269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] PATH_INFO: `/v3/services/image` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:51.844850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:51.846200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:51.854995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:51.855946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:51.856847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 380/785] 10.222.0.20 () {56 vars in 1105 bytes} [Wed Oct 19 21:43:51 2022] GET /identity/v3/services/image => generated 86 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:51.863662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1334ffd3-11f6-4ae0-8b73-65e602e6113a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:51.964848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1334ffd3-11f6-4ae0-8b73-65e602e6113a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:51.997945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:51.998550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:51.998827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:51.999139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:51.999766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] RBAC: Adding query filter params (name=image) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:52.000068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:52.001429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:52.011518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 406/786] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:51 2022] GET /identity/v3/services?name=image => generated 122 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:52.013036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:52.065402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:52.117168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaf855c-19f9-4bfb-9ff4-706f36249150 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:52.150539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd52618c-3425-42d8-9c27-c77f0a70b505 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:52.151738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd52618c-3425-42d8-9c27-c77f0a70b505 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:52.152016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd52618c-3425-42d8-9c27-c77f0a70b505 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:52.152283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd52618c-3425-42d8-9c27-c77f0a70b505 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:52.152922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd52618c-3425-42d8-9c27-c77f0a70b505 None admin] RBAC: Adding query filter params (type=image) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:52.153221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd52618c-3425-42d8-9c27-c77f0a70b505 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:52.154840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd52618c-3425-42d8-9c27-c77f0a70b505 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:52.164411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 381/787] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:52 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:52.165828 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:52.206491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:52.267660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e669167-b506-4376-a46d-a407c6fea949 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:52.302161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2326288-dfae-4577-bfe0-c55096ae01dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:52.303057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2326288-dfae-4577-bfe0-c55096ae01dd None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:52.303537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2326288-dfae-4577-bfe0-c55096ae01dd None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:52.304026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2326288-dfae-4577-bfe0-c55096ae01dd None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:52.304903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2326288-dfae-4577-bfe0-c55096ae01dd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:52.306480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2326288-dfae-4577-bfe0-c55096ae01dd None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:52.414982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 407/788] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:43:52 2022] POST /identity/v3/endpoints => generated 327 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:43:52.516356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdacc252-92bd-4919-a797-12681b3aaa97 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:52.516907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdacc252-92bd-4919-a797-12681b3aaa97 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:52.516907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdacc252-92bd-4919-a797-12681b3aaa97 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:52.518531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 382/789] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:52.524052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c718fa8-4608-41fc-9a54-32dccf5fb1fd None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:52.524338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c718fa8-4608-41fc-9a54-32dccf5fb1fd None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:52.524612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c718fa8-4608-41fc-9a54-32dccf5fb1fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:52.545362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5c718fa8-4608-41fc-9a54-32dccf5fb1fd None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:52.546558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5c718fa8-4608-41fc-9a54-32dccf5fb1fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:52.625384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 408/790] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:52.891897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdacc252-92bd-4919-a797-12681b3aaa97 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:52.979996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c58c8a-34dc-41e7-ae49-f95a5f849202 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:52.980286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c58c8a-34dc-41e7-ae49-f95a5f849202 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:52.980562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c58c8a-34dc-41e7-ae49-f95a5f849202 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:52.982258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 409/791] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:52.987257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3062c-4f71-4599-9e67-64eef2744a5b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:52.987536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3062c-4f71-4599-9e67-64eef2744a5b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:52.987807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3062c-4f71-4599-9e67-64eef2744a5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:53.002032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdacc252-92bd-4919-a797-12681b3aaa97 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:53.005765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-78d3062c-4f71-4599-9e67-64eef2744a5b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:53.006928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-78d3062c-4f71-4599-9e67-64eef2744a5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:53.018447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:53.022006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:53.039837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15b1259-2e7d-407d-863d-cbe30e0cd17d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:53.040653 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b1259-2e7d-407d-863d-cbe30e0cd17d None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:53.041071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b1259-2e7d-407d-863d-cbe30e0cd17d None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:53.041492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15b1259-2e7d-407d-863d-cbe30e0cd17d None admin] PATH_INFO: `/v3/domains/default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:53.042661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15b1259-2e7d-407d-863d-cbe30e0cd17d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:53.044116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b15b1259-2e7d-407d-863d-cbe30e0cd17d None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:53.045673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 383/792] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:53.051304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15b1259-2e7d-407d-863d-cbe30e0cd17d None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:53.076402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 410/793] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:53.154358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15b1259-2e7d-407d-863d-cbe30e0cd17d None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:53.188443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9599703e-24c6-424b-9908-26a6f83cce49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:53.189046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599703e-24c6-424b-9908-26a6f83cce49 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:53.189323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599703e-24c6-424b-9908-26a6f83cce49 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:53.189593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599703e-24c6-424b-9908-26a6f83cce49 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:53.190242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9599703e-24c6-424b-9908-26a6f83cce49 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:53.191692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9599703e-24c6-424b-9908-26a6f83cce49 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:53.269489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 384/794] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:43:53 2022] POST /identity/v3/users => generated 285 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:43:53.345100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d3062c-4f71-4599-9e67-64eef2744a5b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:53.454742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d3062c-4f71-4599-9e67-64eef2744a5b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:53.489716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ddfe15-8018-4496-a281-99a3dd242a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:53.490371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ddfe15-8018-4496-a281-99a3dd242a6f None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:53.490648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ddfe15-8018-4496-a281-99a3dd242a6f None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:53.490915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4ddfe15-8018-4496-a281-99a3dd242a6f None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:53.501744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4ddfe15-8018-4496-a281-99a3dd242a6f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-f4ddfe15-8018-4496-a281-99a3dd242a6f None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:53.506482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:53.507484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:53.508326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 411/795] 10.222.0.20 () {56 vars in 1120 bytes} [Wed Oct 19 21:43:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:53.513683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9599703e-24c6-424b-9908-26a6f83cce49 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:53.556019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:53.616783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9599703e-24c6-424b-9908-26a6f83cce49 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:53.652389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:53.652999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:53.653275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:53.653800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:53.654477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:53.654777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:53.656375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:53.664917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 385/796] 10.222.0.20 () {56 vars in 1115 bytes} [Wed Oct 19 21:43:53 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:53.672684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ddfe15-8018-4496-a281-99a3dd242a6f None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:53.690213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:53.784987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4ddfe15-8018-4496-a281-99a3dd242a6f None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:53.820635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:53.821269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:53.821540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:53.821805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:53.832949 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:53.834405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:53.850446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:53.851506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:53.852398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:53.852398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:53.852398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:53.853273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 412/797] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:53.857367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:53.960521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:53.964097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741790e8-75bc-4df1-a282-c6a0cde320cc None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:53.998376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:53.999005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:53.999285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:53.999555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:54.000207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:54.000502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:54.001867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:54.014730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 386/798] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:54.020402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:54.102307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:54.160217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f63fd4aa-ee0a-4667-94d7-d99c0cc1c3d3 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:54.209824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:54.210662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:54.210888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:54.211190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] PATH_INFO: `/v3/users/nova` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:54.223925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:54.225362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:54.244987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:54.245987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:54.251801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 413/799] 10.222.0.20 () {56 vars in 1093 bytes} [Wed Oct 19 21:43:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:54.253950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:54.364108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb8d785-e644-40ea-813a-0bf3e77bbe34 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:54.399313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:54.399924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:54.400202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:54.400474 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:54.401207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:54.401509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:54.402934 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:54.422073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 387/800] 10.222.0.20 () {56 vars in 1133 bytes} [Wed Oct 19 21:43:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:54.427825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:54.530725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae039d0-a10e-41e9-85e3-e99092b4e62c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:54.553039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:54.565107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612b1f95-c245-4183-8ec3-83e6d0d80614 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:54.565730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b1f95-c245-4183-8ec3-83e6d0d80614 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:54.566116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b1f95-c245-4183-8ec3-83e6d0d80614 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:54.566385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b1f95-c245-4183-8ec3-83e6d0d80614 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:54.576650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612b1f95-c245-4183-8ec3-83e6d0d80614 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:54.578203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612b1f95-c245-4183-8ec3-83e6d0d80614 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-612b1f95-c245-4183-8ec3-83e6d0d80614 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:54.592334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:54.593328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:54.599363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:54.599363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:54.599363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:54.599363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 414/801] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:54.601932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:54.625062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:54.711973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec158e6-589c-41cd-a533-f9643270f8ab None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:54.713364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72477d-4253-4fb5-b5fb-62554eecc16a None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:54.713681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72477d-4253-4fb5-b5fb-62554eecc16a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:54.713956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72477d-4253-4fb5-b5fb-62554eecc16a None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:54.716333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 415/802] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:54.720125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef210327-61ac-4b0d-b096-7bfd3f880a60 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:54.720407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef210327-61ac-4b0d-b096-7bfd3f880a60 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:54.720682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef210327-61ac-4b0d-b096-7bfd3f880a60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:54.739984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ef210327-61ac-4b0d-b096-7bfd3f880a60 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:54.741298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ef210327-61ac-4b0d-b096-7bfd3f880a60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:54.749976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:54.750584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:54.750864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:54.751521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:54.752180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:54.752803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:54.754179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:54.766363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 388/803] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:54.771902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:54.813031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 416/804] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:54.860989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:54.882444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bdb478-2510-40fc-9332-00feadcefa7f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:54.916109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:54.916737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:54.917018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:54.917313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:54.924011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:54.925450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:54.937732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:54.938579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:54.939527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:54.939527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:54.939527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:54.939527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:54.939527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:54.939527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:54.939527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:54.939527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:54.943264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 389/805] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:54.944723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef210327-61ac-4b0d-b096-7bfd3f880a60 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:55.049853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef210327-61ac-4b0d-b096-7bfd3f880a60 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:55.052080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:55.083993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:55.084596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:55.084875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:55.085147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:55.085836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:55.086147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:55.087715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:55.087840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:55.100761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 417/806] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:55.107192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:55.190853 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a53ca-c237-40b7-8a42-31439bad54f8 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:55.212766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41880b44-f158-47dd-9723-d0dd677a8779 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:55.224387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:55.224984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:55.225264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:55.225524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:55.226197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:55.227594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:55.248099 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63776c72-a21e-47f9-9348-88931a1a4543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:55.248756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63776c72-a21e-47f9-9348-88931a1a4543 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:55.249840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63776c72-a21e-47f9-9348-88931a1a4543 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:55.249840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63776c72-a21e-47f9-9348-88931a1a4543 None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/35fc6d90245c4939bc7171201eb883bd/roles/85abe52be1cb4e2fa879120f7cebefbe` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:55.251491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63776c72-a21e-47f9-9348-88931a1a4543 None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=35fc6d90245c4939bc7171201eb883bd, role_id=85abe52be1cb4e2fa879120f7cebefbe)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:55.252929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63776c72-a21e-47f9-9348-88931a1a4543 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:55.268976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.sql.core [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 21:43:55.268976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 21:43:55.268976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [parameters: {'id': 'ce4e982d66594e48aaeb5349b83eccaf', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 21:43:55.268976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=83022) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:43:55.276666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application result.read() Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 19 21:43:55.282136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application yield resource Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application next(self.gen) Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.session.rollback() Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.force_reraise() Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise self.value Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application session.commit() Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 19 21:43:55.283371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._prepare_impl() Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.session.flush() Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._flush(objects) Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application compat.raise_( Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application flush_context.execute() Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rec.execute(self) Oct 19 21:43:55.284294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 19 21:43:55.285152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application result = self._query(query) Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application conn.query(q) Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application result.read() Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 19 21:43:55.285918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry '<>-Default' for key 'project.ixu_project_name_domain_id'") Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [parameters: {'id': 'ce4e982d66594e48aaeb5349b83eccaf', 'name': 'Default', 'domain_id': '<>', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': None, 'is_domain': 1}] Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 19 21:43:55.286792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name Default. Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:55.287609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 128, in post Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_domain( Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 765, in create_domain Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application is_domain_project = self.create_project( Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects acting as domains with the same name: Default. Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 390/807] 10.222.0.20 () {60 vars in 1137 bytes} [Wed Oct 19 21:43:55 2022] POST /identity/v3/domains => generated 190 bytes in 191 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:55.288481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:55.289229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:55.342796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 418/808] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:55 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/35fc6d90245c4939bc7171201eb883bd/roles/85abe52be1cb4e2fa879120f7cebefbe => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 21:43:55.391689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fba4b3-aa35-48a3-8790-0296806c2141 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:55.432053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4c3db0-79fc-4ed1-9267-ffba6bfb8cf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:55.433253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c3db0-79fc-4ed1-9267-ffba6bfb8cf8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:55.433529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c3db0-79fc-4ed1-9267-ffba6bfb8cf8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:55.433869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c3db0-79fc-4ed1-9267-ffba6bfb8cf8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:55.446153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4c3db0-79fc-4ed1-9267-ffba6bfb8cf8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:55.447659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4c3db0-79fc-4ed1-9267-ffba6bfb8cf8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-be4c3db0-79fc-4ed1-9267-ffba6bfb8cf8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:55.464820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:55.465828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:55.466983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:55.466983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:55.466983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:55.466983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 391/809] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:55.472511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63776c72-a21e-47f9-9348-88931a1a4543 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:55.504769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:55.576040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b065cb-f534-4442-acb2-9f7868485dce None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:55.576324 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b065cb-f534-4442-acb2-9f7868485dce None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:55.576600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b065cb-f534-4442-acb2-9f7868485dce None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:55.578066 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 392/810] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:55.583538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd946cc-6443-4247-a08f-bd660817fcca None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:55.583820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd946cc-6443-4247-a08f-bd660817fcca None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:55.584098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd946cc-6443-4247-a08f-bd660817fcca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:55.585054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63776c72-a21e-47f9-9348-88931a1a4543 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:55.602779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3dd946cc-6443-4247-a08f-bd660817fcca None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:55.603940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3dd946cc-6443-4247-a08f-bd660817fcca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:55.624249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:55.624860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:55.625136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:55.625404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:55.626288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:55.626398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:55.627713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:55.651017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 419/811] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:55.658373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:55.677912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 393/812] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:55.946899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:56.028727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:56.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49151a8f-a022-4459-960b-443598ce3d10 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:56.077534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383e8ac0-ea10-43e1-8934-0ca886ae5b21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:56.078162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383e8ac0-ea10-43e1-8934-0ca886ae5b21 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:56.078440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383e8ac0-ea10-43e1-8934-0ca886ae5b21 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:56.078712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-383e8ac0-ea10-43e1-8934-0ca886ae5b21 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:56.088485 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-383e8ac0-ea10-43e1-8934-0ca886ae5b21 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-383e8ac0-ea10-43e1-8934-0ca886ae5b21 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:56.093747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:56.094709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:56.095598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 420/813] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:56.100786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd946cc-6443-4247-a08f-bd660817fcca None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:56.195883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd946cc-6443-4247-a08f-bd660817fcca None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:56.228647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:56.229193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:56.229473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:56.229749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:56.230444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:56.230752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:56.232207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:56.241623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 394/814] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:56.247803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383e8ac0-ea10-43e1-8934-0ca886ae5b21 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:56.345579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-383e8ac0-ea10-43e1-8934-0ca886ae5b21 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:56.377808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068bb346-f84d-420a-98bf-c22443f16b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:56.378419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068bb346-f84d-420a-98bf-c22443f16b67 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:56.378696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068bb346-f84d-420a-98bf-c22443f16b67 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:56.378994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068bb346-f84d-420a-98bf-c22443f16b67 None admin] PATH_INFO: `/v3/users/8969207121454a22802ec413713473b4` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:56.506579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068bb346-f84d-420a-98bf-c22443f16b67 None admin] RBAC: Authorizing `identity:get_user(user_id=8969207121454a22802ec413713473b4)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:56.507951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068bb346-f84d-420a-98bf-c22443f16b67 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:56.509621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 421/815] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:43:56 2022] GET /identity/v3/users/8969207121454a22802ec413713473b4 => generated 285 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:56.515897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:56.564110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:56.617187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5123e1f-b4de-41e5-8bf5-75f3582cc087 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:56.649668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95d4a72c-9cfd-4180-b000-0a6e82e510ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:56.650292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d4a72c-9cfd-4180-b000-0a6e82e510ce None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:56.650568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d4a72c-9cfd-4180-b000-0a6e82e510ce None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:56.650838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d4a72c-9cfd-4180-b000-0a6e82e510ce None admin] PATH_INFO: `/v3/projects/31ec1de105e4454096774bd46d7d55a8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:56.652004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d4a72c-9cfd-4180-b000-0a6e82e510ce None admin] RBAC: Authorizing `identity:get_project(project_id=31ec1de105e4454096774bd46d7d55a8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:56.653689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95d4a72c-9cfd-4180-b000-0a6e82e510ce None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:56.655335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 395/816] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:43:56 2022] GET /identity/v3/projects/31ec1de105e4454096774bd46d7d55a8 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:56.661216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068bb346-f84d-420a-98bf-c22443f16b67 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:56.664074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:56.757674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068bb346-f84d-420a-98bf-c22443f16b67 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:56.789744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65304cb9-c64f-45e6-a50c-dfb362413fa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:56.790403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65304cb9-c64f-45e6-a50c-dfb362413fa3 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:56.790683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65304cb9-c64f-45e6-a50c-dfb362413fa3 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:56.790972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65304cb9-c64f-45e6-a50c-dfb362413fa3 None admin] PATH_INFO: `/v3/projects/31ec1de105e4454096774bd46d7d55a8/users/8969207121454a22802ec413713473b4/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:56.793445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65304cb9-c64f-45e6-a50c-dfb362413fa3 None admin] RBAC: Authorizing `identity:create_grant(project_id=31ec1de105e4454096774bd46d7d55a8, user_id=8969207121454a22802ec413713473b4, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:56.794903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65304cb9-c64f-45e6-a50c-dfb362413fa3 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:56.878999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 422/817] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:43:56 2022] PUT /identity/v3/projects/31ec1de105e4454096774bd46d7d55a8/users/8969207121454a22802ec413713473b4/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 21:43:57.521336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf70be-70fa-4750-b937-714ada6ec8d9 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:57.521735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf70be-70fa-4750-b937-714ada6ec8d9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:57.522013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cf70be-70fa-4750-b937-714ada6ec8d9 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:57.523387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 396/818] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:43:57.528869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c18468-f2aa-4f8b-a762-109189fdf4c6 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:57.529227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c18468-f2aa-4f8b-a762-109189fdf4c6 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:57.529571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c18468-f2aa-4f8b-a762-109189fdf4c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:57.548043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b6c18468-f2aa-4f8b-a762-109189fdf4c6 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:57.549223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b6c18468-f2aa-4f8b-a762-109189fdf4c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:57.603741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:57.614279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 423/819] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:43:57.890640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cf70be-70fa-4750-b937-714ada6ec8d9 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:57.989702 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4cf70be-70fa-4750-b937-714ada6ec8d9 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:58.003761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:58.023777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f62b22-c368-4d3e-b987-7afe52ba845c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:58.024406 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f62b22-c368-4d3e-b987-7afe52ba845c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:58.024686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f62b22-c368-4d3e-b987-7afe52ba845c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:58.024958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4f62b22-c368-4d3e-b987-7afe52ba845c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:58.035878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4f62b22-c368-4d3e-b987-7afe52ba845c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c4f62b22-c368-4d3e-b987-7afe52ba845c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:58.041026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:58.042794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 19 21:43:58.043892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:58.044783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 397/820] 10.222.0.20 () {56 vars in 1120 bytes} [Wed Oct 19 21:43:57 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:58.048619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c18468-f2aa-4f8b-a762-109189fdf4c6 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:58.146247 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c18468-f2aa-4f8b-a762-109189fdf4c6 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:58.181519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:58.182122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:58.182402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:58.182673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:58.183364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:58.183670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:58.185034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:58.192719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 424/821] 10.222.0.20 () {56 vars in 1115 bytes} [Wed Oct 19 21:43:58 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:58.199176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f62b22-c368-4d3e-b987-7afe52ba845c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:58.296402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4f62b22-c368-4d3e-b987-7afe52ba845c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:58.329004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:58.329617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:58.329898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:58.330171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:58.340390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:58.341830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:58.356632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:58.357754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:58.358838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:58.358838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:58.358838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:58.362374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 398/822] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:58.363452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:58.418879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:58.461288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a6d7d-01d3-410e-8f13-a793548dbf4f None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:58.493817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:58.494408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:58.494679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:58.495046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:58.495682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:58.495977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:58.497324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:58.508609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 425/823] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:58.514427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:58.542928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:58.613564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7964f91c-d841-46c1-89f6-cf8b14784977 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:58.645733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:58.646353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:58.646626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:58.646894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] PATH_INFO: `/v3/users/nova` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:58.654412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:58.655858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:43:58.667622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:58.668539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:58.669946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 399/824] 10.222.0.20 () {56 vars in 1093 bytes} [Wed Oct 19 21:43:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:43:58.674380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:58.769616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b27827c-7dc3-4cc1-b03c-e29a75ce50fc None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:58.783282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:58.804911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce210fd-4c2b-4469-be62-03606f8eef44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:58.805509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce210fd-4c2b-4469-be62-03606f8eef44 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:58.805881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce210fd-4c2b-4469-be62-03606f8eef44 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:58.806133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce210fd-4c2b-4469-be62-03606f8eef44 None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:58.806859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce210fd-4c2b-4469-be62-03606f8eef44 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:58.807193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce210fd-4c2b-4469-be62-03606f8eef44 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:58.809232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce210fd-4c2b-4469-be62-03606f8eef44 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:58.827694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 426/825] 10.222.0.20 () {56 vars in 1133 bytes} [Wed Oct 19 21:43:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:58.833475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:58.921144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:58.936240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427fe1a5-d8e7-4501-8c06-1ddc0f6ba003 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:58.969784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ce71ef-fcaf-4742-9a82-c77c037455be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:58.970382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ce71ef-fcaf-4742-9a82-c77c037455be None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:58.970663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ce71ef-fcaf-4742-9a82-c77c037455be None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:58.970933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ce71ef-fcaf-4742-9a82-c77c037455be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:58.992821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf9504-179e-4ce9-8c81-6ccfb82022ee None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:58.993102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf9504-179e-4ce9-8c81-6ccfb82022ee None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:58.993379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf9504-179e-4ce9-8c81-6ccfb82022ee None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:58.994728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 427/826] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:43:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:43:59.000408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97be97b-244a-4c08-869b-baf405e9131b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.000689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97be97b-244a-4c08-869b-baf405e9131b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.000958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97be97b-244a-4c08-869b-baf405e9131b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.018244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a97be97b-244a-4c08-869b-baf405e9131b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:43:59.019471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a97be97b-244a-4c08-869b-baf405e9131b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:43:59.043390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ce71ef-fcaf-4742-9a82-c77c037455be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:59.044939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ce71ef-fcaf-4742-9a82-c77c037455be None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:59.086297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 428/827] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:43:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:43:59.127462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:59.142659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 400/828] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:43:59.148589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97be97b-244a-4c08-869b-baf405e9131b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.242401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97be97b-244a-4c08-869b-baf405e9131b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.243891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:59.276059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.276673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.276946 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.277209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.286920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:59.288371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:43:59.302145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:59.303104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:43:59.304027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:43:59.304027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:43:59.304027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:59.304027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 429/829] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:59.309376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ce71ef-fcaf-4742-9a82-c77c037455be None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.358556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.406814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ce71ef-fcaf-4742-9a82-c77c037455be None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.439441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.440037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.440315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.440594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.441236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:59.441536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:59.442899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:59.453711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 401/830] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:43:59.456818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5545f8d1-9d8a-4bf5-ba9a-39679fda02b3 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.459291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.498115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1be80c-82e3-45eb-8722-32d0ac528bb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.498665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1be80c-82e3-45eb-8722-32d0ac528bb1 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.498943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1be80c-82e3-45eb-8722-32d0ac528bb1 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.499237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1be80c-82e3-45eb-8722-32d0ac528bb1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.509245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1be80c-82e3-45eb-8722-32d0ac528bb1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:43:59.510437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-ad1be80c-82e3-45eb-8722-32d0ac528bb1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:43:59.513966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:59.514886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:43:59.519596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 430/831] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:59.521157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1be80c-82e3-45eb-8722-32d0ac528bb1 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.537276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:59.563925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e01dc71-ec1b-44b0-979b-3a35f7c56546 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.600861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.601487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.601760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.602026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.608865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:59.610310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:59.621931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1be80c-82e3-45eb-8722-32d0ac528bb1 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:43:59.622147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:43:59.623000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:43:59.623902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:43:59.623902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:43:59.623902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:43:59.623902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:43:59.623902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:43:59.623902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:43:59.623902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:43:59.623902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:43:59.628257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 402/832] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:43:59.629027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.665292 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.665984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.667393 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.667702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.668400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:59.668709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:59.671194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:59.681269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 431/833] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:59.687432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.735508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e94efbf-f936-4818-b7ce-3784bb8def69 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.742722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:59.761623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:43:59.769066 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.769665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.769943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.770214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.771047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:59.771353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:59.772783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:59.783604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9d3038-2363-4d66-a997-d5c2b26bdab4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.787604 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 403/834] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:59.793767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.820870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238f4735-567b-408e-ac2a-d1488915db49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.821493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238f4735-567b-408e-ac2a-d1488915db49 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.821767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238f4735-567b-408e-ac2a-d1488915db49 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.822032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238f4735-567b-408e-ac2a-d1488915db49 None admin] PATH_INFO: `/v3/users/8969207121454a22802ec413713473b4` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.823233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238f4735-567b-408e-ac2a-d1488915db49 None admin] RBAC: Authorizing `identity:get_user(user_id=8969207121454a22802ec413713473b4)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:59.824643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238f4735-567b-408e-ac2a-d1488915db49 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:59.826210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 432/835] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/users/8969207121454a22802ec413713473b4 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:59.831721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238f4735-567b-408e-ac2a-d1488915db49 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:43:59.891340 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b2d22-13ed-4886-9aa8-c6f6143f477d None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.925740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.926339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.926595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.926859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.927609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] RBAC: Adding query filter params (role.id=85abe52be1cb4e2fa879120f7cebefbe, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=35fc6d90245c4939bc7171201eb883bd) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:43:59.927913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:43:59.928094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238f4735-567b-408e-ac2a-d1488915db49 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:43:59.929325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:43:59.934980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 404/836] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/role_assignments?user.id=35fc6d90245c4939bc7171201eb883bd&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=85abe52be1cb4e2fa879120f7cebefbe => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:43:59.971997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe2f47b-3f88-46c5-8817-b7fb03529664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:43:59.972619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2f47b-3f88-46c5-8817-b7fb03529664 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:43:59.972894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2f47b-3f88-46c5-8817-b7fb03529664 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:43:59.973162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2f47b-3f88-46c5-8817-b7fb03529664 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:43:59.989352 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:00.045943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe2f47b-3f88-46c5-8817-b7fb03529664 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:00.047363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe2f47b-3f88-46c5-8817-b7fb03529664 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:00.148924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 433/837] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:43:59 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 19 21:44:00.155285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:00.186081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:00.257453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e639bc-ce28-41dd-8127-53b34cfc1498 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:00.291486 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2fc9f46-e9e0-4399-8f06-63a98ca6ba97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:00.292689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fc9f46-e9e0-4399-8f06-63a98ca6ba97 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:00.292964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fc9f46-e9e0-4399-8f06-63a98ca6ba97 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:00.293236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2fc9f46-e9e0-4399-8f06-63a98ca6ba97 None admin] PATH_INFO: `/v3/projects/31ec1de105e4454096774bd46d7d55a8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:00.294360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fc9f46-e9e0-4399-8f06-63a98ca6ba97 None admin] RBAC: Authorizing `identity:get_project(project_id=31ec1de105e4454096774bd46d7d55a8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:00.295826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2fc9f46-e9e0-4399-8f06-63a98ca6ba97 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:00.297509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 405/838] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:44:00 2022] GET /identity/v3/projects/31ec1de105e4454096774bd46d7d55a8 => generated 311 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:00.303777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe2f47b-3f88-46c5-8817-b7fb03529664 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:00.352374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:00.406747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe2f47b-3f88-46c5-8817-b7fb03529664 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:00.440094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4275c0fc-c252-42d2-97e2-b2c5cc0af227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:00.440698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4275c0fc-c252-42d2-97e2-b2c5cc0af227 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:00.440975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4275c0fc-c252-42d2-97e2-b2c5cc0af227 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:00.441253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4275c0fc-c252-42d2-97e2-b2c5cc0af227 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:00.441980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4275c0fc-c252-42d2-97e2-b2c5cc0af227 None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=31ec1de105e4454096774bd46d7d55a8, user.id=8969207121454a22802ec413713473b4) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:44:00.442289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4275c0fc-c252-42d2-97e2-b2c5cc0af227 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:00.443799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4275c0fc-c252-42d2-97e2-b2c5cc0af227 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:00.450065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 434/839] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:44:00 2022] GET /identity/v3/role_assignments?user.id=8969207121454a22802ec413713473b4&scope.project.id=31ec1de105e4454096774bd46d7d55a8&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:02.583071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a45aee-d6cd-4a25-a072-7c19307c9142 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:02.583648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a45aee-d6cd-4a25-a072-7c19307c9142 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:02.583648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a45aee-d6cd-4a25-a072-7c19307c9142 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:02.584981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 406/840] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:02.590851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d550cd0-69c6-4aa8-ae75-1fc9f8fe23a6 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:02.591296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d550cd0-69c6-4aa8-ae75-1fc9f8fe23a6 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:02.591432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d550cd0-69c6-4aa8-ae75-1fc9f8fe23a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:02.608869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5d550cd0-69c6-4aa8-ae75-1fc9f8fe23a6 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:02.610027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5d550cd0-69c6-4aa8-ae75-1fc9f8fe23a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:02.673358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 435/841] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:02.941046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a45aee-d6cd-4a25-a072-7c19307c9142 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:03.035208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:03.039802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a45aee-d6cd-4a25-a072-7c19307c9142 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:03.072494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62396dfc-1bfd-4027-bd44-877eea3b3534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:03.073742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62396dfc-1bfd-4027-bd44-877eea3b3534 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:03.074013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62396dfc-1bfd-4027-bd44-877eea3b3534 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:03.074280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62396dfc-1bfd-4027-bd44-877eea3b3534 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:03.075482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62396dfc-1bfd-4027-bd44-877eea3b3534 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:03.076910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62396dfc-1bfd-4027-bd44-877eea3b3534 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:03.078449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 407/842] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:44:02 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:03.084160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d550cd0-69c6-4aa8-ae75-1fc9f8fe23a6 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:03.122772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:03.179151 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d550cd0-69c6-4aa8-ae75-1fc9f8fe23a6 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:03.212442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2e920a-a1ee-4fe9-81c6-8ec6ca2f68bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:03.216404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2e920a-a1ee-4fe9-81c6-8ec6ca2f68bd None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:03.216404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2e920a-a1ee-4fe9-81c6-8ec6ca2f68bd None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:03.216404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2e920a-a1ee-4fe9-81c6-8ec6ca2f68bd None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:03.216404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2e920a-a1ee-4fe9-81c6-8ec6ca2f68bd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:03.216404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2e920a-a1ee-4fe9-81c6-8ec6ca2f68bd None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:03.297953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 436/843] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:44:03 2022] POST /identity/v3/users => generated 286 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:44:05.436332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6c620-f84c-4a3e-9ee5-32e0b5d9f9a0 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:05.436889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6c620-f84c-4a3e-9ee5-32e0b5d9f9a0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:05.436889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd6c620-f84c-4a3e-9ee5-32e0b5d9f9a0 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:05.438240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 408/844] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:05.443381 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67372c9-c615-4982-9d5f-efc688b9f923 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:05.443800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67372c9-c615-4982-9d5f-efc688b9f923 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:05.443940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67372c9-c615-4982-9d5f-efc688b9f923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:05.462058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f67372c9-c615-4982-9d5f-efc688b9f923 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:05.463269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f67372c9-c615-4982-9d5f-efc688b9f923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:05.526756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 437/845] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:05.802530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6c620-f84c-4a3e-9ee5-32e0b5d9f9a0 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:05.900246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd6c620-f84c-4a3e-9ee5-32e0b5d9f9a0 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:05.933404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e958bf9-86e3-49aa-af1a-da7f9b9aaabc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:05.934013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e958bf9-86e3-49aa-af1a-da7f9b9aaabc None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:05.934290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e958bf9-86e3-49aa-af1a-da7f9b9aaabc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:05.934561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e958bf9-86e3-49aa-af1a-da7f9b9aaabc None admin] PATH_INFO: `/v3/roles/3cb31ad012324a57acd3a4bd258d1fb1` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:05.936122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e958bf9-86e3-49aa-af1a-da7f9b9aaabc None admin] RBAC: Authorizing `identity:get_role(role_id=3cb31ad012324a57acd3a4bd258d1fb1)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:05.937497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e958bf9-86e3-49aa-af1a-da7f9b9aaabc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:05.938797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 409/846] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:44:05 2022] GET /identity/v3/roles/3cb31ad012324a57acd3a4bd258d1fb1 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:05.944472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67372c9-c615-4982-9d5f-efc688b9f923 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:05.968452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:06.039836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67372c9-c615-4982-9d5f-efc688b9f923 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:06.072330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2460f31a-7a17-4cce-9c10-b1528bfedf86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:06.073533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2460f31a-7a17-4cce-9c10-b1528bfedf86 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:06.073816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2460f31a-7a17-4cce-9c10-b1528bfedf86 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:06.074091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2460f31a-7a17-4cce-9c10-b1528bfedf86 None admin] PATH_INFO: `/v3/users/0a1a75e92fb94e41bf8d7e115b4501c8` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:06.086464 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2460f31a-7a17-4cce-9c10-b1528bfedf86 None admin] RBAC: Authorizing `identity:get_user(user_id=0a1a75e92fb94e41bf8d7e115b4501c8)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:06.087924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2460f31a-7a17-4cce-9c10-b1528bfedf86 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:06.089511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 438/847] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:44:05 2022] GET /identity/v3/users/0a1a75e92fb94e41bf8d7e115b4501c8 => generated 286 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:06.095128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e958bf9-86e3-49aa-af1a-da7f9b9aaabc None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:06.128426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:06.192693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e958bf9-86e3-49aa-af1a-da7f9b9aaabc None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:06.225269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12a2a78-71e3-4b16-ac1a-858edac63c7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:06.226603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a2a78-71e3-4b16-ac1a-858edac63c7f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:06.226603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a2a78-71e3-4b16-ac1a-858edac63c7f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:06.226603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a2a78-71e3-4b16-ac1a-858edac63c7f None admin] PATH_INFO: `/v3/projects/31ec1de105e4454096774bd46d7d55a8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:06.227619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a2a78-71e3-4b16-ac1a-858edac63c7f None admin] RBAC: Authorizing `identity:get_project(project_id=31ec1de105e4454096774bd46d7d55a8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:06.229040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a2a78-71e3-4b16-ac1a-858edac63c7f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:06.230665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 410/848] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:44:06 2022] GET /identity/v3/projects/31ec1de105e4454096774bd46d7d55a8 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:06.236678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2460f31a-7a17-4cce-9c10-b1528bfedf86 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:06.315493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:06.332187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2460f31a-7a17-4cce-9c10-b1528bfedf86 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:06.364691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864146f8-c02a-46a1-8ed8-5cb2164c3274 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:06.365334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864146f8-c02a-46a1-8ed8-5cb2164c3274 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:06.365609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864146f8-c02a-46a1-8ed8-5cb2164c3274 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:06.365881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864146f8-c02a-46a1-8ed8-5cb2164c3274 None admin] PATH_INFO: `/v3/projects/31ec1de105e4454096774bd46d7d55a8/users/0a1a75e92fb94e41bf8d7e115b4501c8/roles/3cb31ad012324a57acd3a4bd258d1fb1` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:06.368024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864146f8-c02a-46a1-8ed8-5cb2164c3274 None admin] RBAC: Authorizing `identity:create_grant(project_id=31ec1de105e4454096774bd46d7d55a8, user_id=0a1a75e92fb94e41bf8d7e115b4501c8, role_id=3cb31ad012324a57acd3a4bd258d1fb1)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:06.369443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864146f8-c02a-46a1-8ed8-5cb2164c3274 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:06.445890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 439/849] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:44:06 2022] PUT /identity/v3/projects/31ec1de105e4454096774bd46d7d55a8/users/0a1a75e92fb94e41bf8d7e115b4501c8/roles/3cb31ad012324a57acd3a4bd258d1fb1 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:44:08.530441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc14605-6cd9-4f62-9c38-9ebf300e88ae None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:08.530991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc14605-6cd9-4f62-9c38-9ebf300e88ae None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:08.530991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc14605-6cd9-4f62-9c38-9ebf300e88ae None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:08.532319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 411/850] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:08.537530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5dc13-e0ad-414b-89ce-8ffb7f1d7cfc None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:08.537947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5dc13-e0ad-414b-89ce-8ffb7f1d7cfc None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:08.538083 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5dc13-e0ad-414b-89ce-8ffb7f1d7cfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:08.556117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-00f5dc13-e0ad-414b-89ce-8ffb7f1d7cfc None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:08.557851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-00f5dc13-e0ad-414b-89ce-8ffb7f1d7cfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:08.620892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 440/851] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:08.887229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc14605-6cd9-4f62-9c38-9ebf300e88ae None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:08.985375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc14605-6cd9-4f62-9c38-9ebf300e88ae None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:08.986848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:09.017636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db03feba-f7f0-47d9-abda-dc77ddc22a03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:09.018248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db03feba-f7f0-47d9-abda-dc77ddc22a03 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:09.018530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db03feba-f7f0-47d9-abda-dc77ddc22a03 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:09.018810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db03feba-f7f0-47d9-abda-dc77ddc22a03 None admin] PATH_INFO: `/v3/roles/3cb31ad012324a57acd3a4bd258d1fb1` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:09.020234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db03feba-f7f0-47d9-abda-dc77ddc22a03 None admin] RBAC: Authorizing `identity:get_role(role_id=3cb31ad012324a57acd3a4bd258d1fb1)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:09.021655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db03feba-f7f0-47d9-abda-dc77ddc22a03 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:09.022964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 412/852] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:44:08 2022] GET /identity/v3/roles/3cb31ad012324a57acd3a4bd258d1fb1 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:09.029514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5dc13-e0ad-414b-89ce-8ffb7f1d7cfc None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:09.095642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:09.124007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5dc13-e0ad-414b-89ce-8ffb7f1d7cfc None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:09.162026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0f4d22-4671-4bc7-b98e-ba352d79c398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:09.162681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0f4d22-4671-4bc7-b98e-ba352d79c398 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:09.162981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0f4d22-4671-4bc7-b98e-ba352d79c398 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:09.163285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0f4d22-4671-4bc7-b98e-ba352d79c398 None admin] PATH_INFO: `/v3/users/0a1a75e92fb94e41bf8d7e115b4501c8` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:09.164401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0f4d22-4671-4bc7-b98e-ba352d79c398 None admin] RBAC: Authorizing `identity:get_user(user_id=0a1a75e92fb94e41bf8d7e115b4501c8)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:09.165945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0f4d22-4671-4bc7-b98e-ba352d79c398 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:09.168163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 441/853] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:44:09 2022] GET /identity/v3/users/0a1a75e92fb94e41bf8d7e115b4501c8 => generated 286 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:09.173911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db03feba-f7f0-47d9-abda-dc77ddc22a03 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:09.272335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db03feba-f7f0-47d9-abda-dc77ddc22a03 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:09.305814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48733f49-1763-42c7-9bdd-0ba252a28896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:09.306409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48733f49-1763-42c7-9bdd-0ba252a28896 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:09.306689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48733f49-1763-42c7-9bdd-0ba252a28896 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:09.306977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48733f49-1763-42c7-9bdd-0ba252a28896 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:09.344894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:09.372092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48733f49-1763-42c7-9bdd-0ba252a28896 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:09.373792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48733f49-1763-42c7-9bdd-0ba252a28896 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:09.468268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 413/854] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:44:09 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:44:09.474112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0f4d22-4671-4bc7-b98e-ba352d79c398 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:09.570678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0f4d22-4671-4bc7-b98e-ba352d79c398 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:09.602395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d29495-5527-4248-a8e3-39b8ae947947 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:09.603047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d29495-5527-4248-a8e3-39b8ae947947 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:09.603329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d29495-5527-4248-a8e3-39b8ae947947 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:09.603602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d29495-5527-4248-a8e3-39b8ae947947 None admin] PATH_INFO: `/v3/projects/31ec1de105e4454096774bd46d7d55a8` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:09.607101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d29495-5527-4248-a8e3-39b8ae947947 None admin] RBAC: Authorizing `identity:get_project(project_id=31ec1de105e4454096774bd46d7d55a8)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:09.609098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d29495-5527-4248-a8e3-39b8ae947947 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:09.611764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 442/855] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:44:09 2022] GET /identity/v3/projects/31ec1de105e4454096774bd46d7d55a8 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:09.622564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48733f49-1763-42c7-9bdd-0ba252a28896 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:09.688794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:09.718207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48733f49-1763-42c7-9bdd-0ba252a28896 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:09.750041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01e92ee-34e7-4c64-b40a-0ac1e4adb595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:09.750622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e92ee-34e7-4c64-b40a-0ac1e4adb595 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:09.750895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e92ee-34e7-4c64-b40a-0ac1e4adb595 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:09.751190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01e92ee-34e7-4c64-b40a-0ac1e4adb595 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:09.751909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01e92ee-34e7-4c64-b40a-0ac1e4adb595 None admin] RBAC: Adding query filter params (role.id=3cb31ad012324a57acd3a4bd258d1fb1, scope.project.id=31ec1de105e4454096774bd46d7d55a8, user.id=0a1a75e92fb94e41bf8d7e115b4501c8) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:44:09.752213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01e92ee-34e7-4c64-b40a-0ac1e4adb595 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:09.753622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b01e92ee-34e7-4c64-b40a-0ac1e4adb595 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:09.759095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 414/856] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:44:09 2022] GET /identity/v3/role_assignments?user.id=0a1a75e92fb94e41bf8d7e115b4501c8&scope.project.id=31ec1de105e4454096774bd46d7d55a8&role.id=3cb31ad012324a57acd3a4bd258d1fb1 => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:11.887087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228c013-aa43-4dd1-b1da-0adf2aff95d7 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:11.887624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228c013-aa43-4dd1-b1da-0adf2aff95d7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:11.887722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4228c013-aa43-4dd1-b1da-0adf2aff95d7 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:11.889026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 443/857] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:11.894194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf34fe-1b57-4da0-ba2c-219ec0ba4b88 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:11.894598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf34fe-1b57-4da0-ba2c-219ec0ba4b88 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:11.894736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bcf34fe-1b57-4da0-ba2c-219ec0ba4b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:11.913133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-2bcf34fe-1b57-4da0-ba2c-219ec0ba4b88 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:11.914301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2bcf34fe-1b57-4da0-ba2c-219ec0ba4b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:11.978626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 415/858] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:12.244006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4228c013-aa43-4dd1-b1da-0adf2aff95d7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:12.255136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:12.340799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4228c013-aa43-4dd1-b1da-0adf2aff95d7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:12.373063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b277a65c-a7af-4471-811c-e1b73249517a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:12.373664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277a65c-a7af-4471-811c-e1b73249517a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:12.373938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277a65c-a7af-4471-811c-e1b73249517a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:12.374206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b277a65c-a7af-4471-811c-e1b73249517a None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:12.375310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b277a65c-a7af-4471-811c-e1b73249517a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:12.376721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b277a65c-a7af-4471-811c-e1b73249517a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:12.378234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 444/859] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:44:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:12.383981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcf34fe-1b57-4da0-ba2c-219ec0ba4b88 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:12.442509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:12.477614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bcf34fe-1b57-4da0-ba2c-219ec0ba4b88 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:12.510059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4af700-cc09-40f6-a024-2ab729fc71f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:12.511173 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4af700-cc09-40f6-a024-2ab729fc71f0 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:12.511441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4af700-cc09-40f6-a024-2ab729fc71f0 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:12.511704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4af700-cc09-40f6-a024-2ab729fc71f0 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:12.512342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4af700-cc09-40f6-a024-2ab729fc71f0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:12.513750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4af700-cc09-40f6-a024-2ab729fc71f0 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:12.590036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 416/860] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:44:12 2022] POST /identity/v3/projects => generated 311 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:44:14.728070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8559ad1a-1a18-4c72-aee4-f40767eb5cfd None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:14.728602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8559ad1a-1a18-4c72-aee4-f40767eb5cfd None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:14.728602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8559ad1a-1a18-4c72-aee4-f40767eb5cfd None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:14.730413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 445/861] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:14.735541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2685e43-2089-4dea-922a-c647b00befad None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:14.735962 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2685e43-2089-4dea-922a-c647b00befad None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:14.736115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2685e43-2089-4dea-922a-c647b00befad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:14.754264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a2685e43-2089-4dea-922a-c647b00befad None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:14.755470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a2685e43-2089-4dea-922a-c647b00befad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:14.819559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 417/862] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:15.085380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8559ad1a-1a18-4c72-aee4-f40767eb5cfd None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:15.159682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:15.186626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8559ad1a-1a18-4c72-aee4-f40767eb5cfd None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:15.219527 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25a6162-d8aa-4dd9-88c1-779fe4585d48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:15.220134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a6162-d8aa-4dd9-88c1-779fe4585d48 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:15.220408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a6162-d8aa-4dd9-88c1-779fe4585d48 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:15.220676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25a6162-d8aa-4dd9-88c1-779fe4585d48 None admin] PATH_INFO: `/v3/domains/default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:15.221743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a6162-d8aa-4dd9-88c1-779fe4585d48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:15.223184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25a6162-d8aa-4dd9-88c1-779fe4585d48 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:15.224708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 446/863] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:44:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:15.231115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2685e43-2089-4dea-922a-c647b00befad None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:15.274127 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:15.327040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2685e43-2089-4dea-922a-c647b00befad None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:15.360071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14686a1-5cb8-40d4-88f6-de2ac68c1955 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:15.360607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14686a1-5cb8-40d4-88f6-de2ac68c1955 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:15.360886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14686a1-5cb8-40d4-88f6-de2ac68c1955 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:15.361161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14686a1-5cb8-40d4-88f6-de2ac68c1955 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:15.361809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14686a1-5cb8-40d4-88f6-de2ac68c1955 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:15.363262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14686a1-5cb8-40d4-88f6-de2ac68c1955 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:15.463321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 418/864] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:44:15 2022] POST /identity/v3/users => generated 286 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:44:17.592547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6c646b-0c8a-4105-84ff-555725151bdf None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:17.593110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6c646b-0c8a-4105-84ff-555725151bdf None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:17.593110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6c646b-0c8a-4105-84ff-555725151bdf None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:17.594421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 447/865] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:44:17.599695 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016e243-68b9-4ac6-bcd4-fd9436d010bb None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:17.600082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016e243-68b9-4ac6-bcd4-fd9436d010bb None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:17.600243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b016e243-68b9-4ac6-bcd4-fd9436d010bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:17.618305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b016e243-68b9-4ac6-bcd4-fd9436d010bb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:17.619502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b016e243-68b9-4ac6-bcd4-fd9436d010bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:17.687681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 419/866] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:17.951956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6c646b-0c8a-4105-84ff-555725151bdf None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:18.049955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6c646b-0c8a-4105-84ff-555725151bdf None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:18.082029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c430f280-b624-4d63-9b81-73dcb96fb9f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:18.082634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c430f280-b624-4d63-9b81-73dcb96fb9f7 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:18.082911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c430f280-b624-4d63-9b81-73dcb96fb9f7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:18.083217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c430f280-b624-4d63-9b81-73dcb96fb9f7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:18.093453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c430f280-b624-4d63-9b81-73dcb96fb9f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-c430f280-b624-4d63-9b81-73dcb96fb9f7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:44:18.098180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:44:18.099326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:44:18.100406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 448/867] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:44:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:44:18.105714 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016e243-68b9-4ac6-bcd4-fd9436d010bb None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:18.134278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:18.201132 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b016e243-68b9-4ac6-bcd4-fd9436d010bb None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:18.235832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:18.236372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:18.236645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:18.236914 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:18.237572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:44:18.237870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:18.239257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:18.247149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 420/868] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:44:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:18.252918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c430f280-b624-4d63-9b81-73dcb96fb9f7 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:18.257208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:18.350646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c430f280-b624-4d63-9b81-73dcb96fb9f7 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:18.382452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c01842b-3634-435f-bd12-9a35ca10576d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:18.383091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01842b-3634-435f-bd12-9a35ca10576d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:18.383371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01842b-3634-435f-bd12-9a35ca10576d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:18.383645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c01842b-3634-435f-bd12-9a35ca10576d None admin] PATH_INFO: `/v3/users/0dd9e4ae516a4036b42c946be914a392` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:18.460797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c01842b-3634-435f-bd12-9a35ca10576d None admin] RBAC: Authorizing `identity:get_user(user_id=0dd9e4ae516a4036b42c946be914a392)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:18.462341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c01842b-3634-435f-bd12-9a35ca10576d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:18.464643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 449/869] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:44:18 2022] GET /identity/v3/users/0dd9e4ae516a4036b42c946be914a392 => generated 286 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:18.470390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:18.559644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:18.565938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98abc34f-033e-44cd-be73-3e32ddb44ab9 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:18.598810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc936f42-3ae6-49a0-940f-35a76a824526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:18.599444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc936f42-3ae6-49a0-940f-35a76a824526 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:18.599718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc936f42-3ae6-49a0-940f-35a76a824526 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:18.600241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc936f42-3ae6-49a0-940f-35a76a824526 None admin] PATH_INFO: `/v3/projects/7ec521bc938f4778a52fe8b514203ecb` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:18.601069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc936f42-3ae6-49a0-940f-35a76a824526 None admin] RBAC: Authorizing `identity:get_project(project_id=7ec521bc938f4778a52fe8b514203ecb)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:18.602487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc936f42-3ae6-49a0-940f-35a76a824526 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:18.604132 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 421/870] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:44:18 2022] GET /identity/v3/projects/7ec521bc938f4778a52fe8b514203ecb => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:18.609975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c01842b-3634-435f-bd12-9a35ca10576d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:18.669410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:18.707123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c01842b-3634-435f-bd12-9a35ca10576d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:18.739274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea2e6aa-8c2f-4898-9fad-8be7025cae42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:18.739911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea2e6aa-8c2f-4898-9fad-8be7025cae42 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:18.740189 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea2e6aa-8c2f-4898-9fad-8be7025cae42 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:18.740462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea2e6aa-8c2f-4898-9fad-8be7025cae42 None admin] PATH_INFO: `/v3/projects/7ec521bc938f4778a52fe8b514203ecb/users/0dd9e4ae516a4036b42c946be914a392/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:18.742345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea2e6aa-8c2f-4898-9fad-8be7025cae42 None admin] RBAC: Authorizing `identity:create_grant(project_id=7ec521bc938f4778a52fe8b514203ecb, user_id=0dd9e4ae516a4036b42c946be914a392, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:18.743795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea2e6aa-8c2f-4898-9fad-8be7025cae42 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:18.800655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 450/871] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:44:18 2022] PUT /identity/v3/projects/7ec521bc938f4778a52fe8b514203ecb/users/0dd9e4ae516a4036b42c946be914a392/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:44:20.887319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9857271-1db0-4119-9283-82e31c984f70 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:20.887863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9857271-1db0-4119-9283-82e31c984f70 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:20.887863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9857271-1db0-4119-9283-82e31c984f70 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:20.889178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 422/872] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:20.894370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0e7552-6b1a-41d2-99d4-bb308bd14dc4 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:20.894742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0e7552-6b1a-41d2-99d4-bb308bd14dc4 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:20.894917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0e7552-6b1a-41d2-99d4-bb308bd14dc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:20.913157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-9e0e7552-6b1a-41d2-99d4-bb308bd14dc4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:20.914348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9e0e7552-6b1a-41d2-99d4-bb308bd14dc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:20.976512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 451/873] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:44:21.247131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9857271-1db0-4119-9283-82e31c984f70 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:21.344443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9857271-1db0-4119-9283-82e31c984f70 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:21.376448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138dedc4-f2e9-40ec-9d2e-ac690889e36f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:21.377056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138dedc4-f2e9-40ec-9d2e-ac690889e36f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:21.377332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138dedc4-f2e9-40ec-9d2e-ac690889e36f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:21.377599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138dedc4-f2e9-40ec-9d2e-ac690889e36f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:21.388318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-138dedc4-f2e9-40ec-9d2e-ac690889e36f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-138dedc4-f2e9-40ec-9d2e-ac690889e36f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:44:21.393192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:44:21.394253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:44:21.395302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 423/874] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:44:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:44:21.400307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0e7552-6b1a-41d2-99d4-bb308bd14dc4 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:21.450374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:21.494248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0e7552-6b1a-41d2-99d4-bb308bd14dc4 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:21.527304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:21.527832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:21.528105 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:21.528369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:21.529011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:44:21.529561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:21.530901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:21.538481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 452/875] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:44:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:21.544849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138dedc4-f2e9-40ec-9d2e-ac690889e36f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:21.569348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:21.641402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138dedc4-f2e9-40ec-9d2e-ac690889e36f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:21.674837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085028c4-7c72-4216-8a2e-daac6786a7a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:21.675468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085028c4-7c72-4216-8a2e-daac6786a7a5 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:21.675745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085028c4-7c72-4216-8a2e-daac6786a7a5 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:21.676016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085028c4-7c72-4216-8a2e-daac6786a7a5 None admin] PATH_INFO: `/v3/users/0dd9e4ae516a4036b42c946be914a392` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:21.677130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085028c4-7c72-4216-8a2e-daac6786a7a5 None admin] RBAC: Authorizing `identity:get_user(user_id=0dd9e4ae516a4036b42c946be914a392)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:21.678561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085028c4-7c72-4216-8a2e-daac6786a7a5 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:21.680129 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 424/876] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:44:21 2022] GET /identity/v3/users/0dd9e4ae516a4036b42c946be914a392 => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:21.685837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:21.782687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33828914-f3d4-492a-91d5-18808bc5e905 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:21.796712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:21.815818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99edd78-596a-47f0-8854-62af03607057 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:21.816410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99edd78-596a-47f0-8854-62af03607057 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:21.816685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99edd78-596a-47f0-8854-62af03607057 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:21.816955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99edd78-596a-47f0-8854-62af03607057 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:21.880913 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99edd78-596a-47f0-8854-62af03607057 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:21.882348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99edd78-596a-47f0-8854-62af03607057 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:21.975134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 453/877] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:44:21 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:44:21.981681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085028c4-7c72-4216-8a2e-daac6786a7a5 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:22.068244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:22.077768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085028c4-7c72-4216-8a2e-daac6786a7a5 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:22.109804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eac1c37-ae22-43d8-8272-a771dc10863c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:22.110744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eac1c37-ae22-43d8-8272-a771dc10863c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:22.111047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eac1c37-ae22-43d8-8272-a771dc10863c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:22.111318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eac1c37-ae22-43d8-8272-a771dc10863c None admin] PATH_INFO: `/v3/projects/7ec521bc938f4778a52fe8b514203ecb` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:22.112421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eac1c37-ae22-43d8-8272-a771dc10863c None admin] RBAC: Authorizing `identity:get_project(project_id=7ec521bc938f4778a52fe8b514203ecb)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:22.113838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eac1c37-ae22-43d8-8272-a771dc10863c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:22.115449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 425/878] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:44:21 2022] GET /identity/v3/projects/7ec521bc938f4778a52fe8b514203ecb => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:22.121604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99edd78-596a-47f0-8854-62af03607057 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:22.136476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:22.217809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99edd78-596a-47f0-8854-62af03607057 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:22.250244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d297298d-efbc-479c-bb8b-a2c913175953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:22.250833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d297298d-efbc-479c-bb8b-a2c913175953 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:22.251142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d297298d-efbc-479c-bb8b-a2c913175953 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:22.251419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d297298d-efbc-479c-bb8b-a2c913175953 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:22.252140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d297298d-efbc-479c-bb8b-a2c913175953 None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=7ec521bc938f4778a52fe8b514203ecb, user.id=0dd9e4ae516a4036b42c946be914a392) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:44:22.252448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d297298d-efbc-479c-bb8b-a2c913175953 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:22.253867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d297298d-efbc-479c-bb8b-a2c913175953 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:22.259421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 454/879] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:44:22 2022] GET /identity/v3/role_assignments?user.id=0dd9e4ae516a4036b42c946be914a392&scope.project.id=7ec521bc938f4778a52fe8b514203ecb&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:44:24.381863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3c171-c681-447b-8c06-ec8232cfd68b None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:24.382394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3c171-c681-447b-8c06-ec8232cfd68b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:24.382394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a3c171-c681-447b-8c06-ec8232cfd68b None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:24.383756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 426/880] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:24.389489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2e02bd-0d80-4f4c-94ee-bb3e6cd8fb3b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:24.389921 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2e02bd-0d80-4f4c-94ee-bb3e6cd8fb3b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:24.390020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd2e02bd-0d80-4f4c-94ee-bb3e6cd8fb3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:24.408582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-cd2e02bd-0d80-4f4c-94ee-bb3e6cd8fb3b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:24.409750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cd2e02bd-0d80-4f4c-94ee-bb3e6cd8fb3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:24.473463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 455/881] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:24.741783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3c171-c681-447b-8c06-ec8232cfd68b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:24.840623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a3c171-c681-447b-8c06-ec8232cfd68b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:24.872905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daaa26fe-c1aa-4c68-8df6-b1ff2b6bfd79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:24.873496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaa26fe-c1aa-4c68-8df6-b1ff2b6bfd79 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:24.873772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaa26fe-c1aa-4c68-8df6-b1ff2b6bfd79 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:24.874042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaa26fe-c1aa-4c68-8df6-b1ff2b6bfd79 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:24.874716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daaa26fe-c1aa-4c68-8df6-b1ff2b6bfd79 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:24.876100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daaa26fe-c1aa-4c68-8df6-b1ff2b6bfd79 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:24.959907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 427/882] 10.222.0.20 () {60 vars in 1138 bytes} [Wed Oct 19 21:44:24 2022] POST /identity/v3/domains => generated 267 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:44:27.102478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f1e123-1057-4993-86cf-3b8485c6ea3c None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:27.103029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f1e123-1057-4993-86cf-3b8485c6ea3c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:27.103029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f1e123-1057-4993-86cf-3b8485c6ea3c None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:27.104434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 456/883] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:27.109524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993c9c84-7e41-42ca-9cc5-1b5aa0caee35 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:27.109903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993c9c84-7e41-42ca-9cc5-1b5aa0caee35 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:27.110071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993c9c84-7e41-42ca-9cc5-1b5aa0caee35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:27.128204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-993c9c84-7e41-42ca-9cc5-1b5aa0caee35 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:27.129382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-993c9c84-7e41-42ca-9cc5-1b5aa0caee35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:27.142598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:27.193823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 428/884] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:27.465467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f1e123-1057-4993-86cf-3b8485c6ea3c None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:27.476246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:27.563329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f1e123-1057-4993-86cf-3b8485c6ea3c None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:27.594417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a785e13-e2c6-45b6-9348-03864a9a7015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:27.595048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785e13-e2c6-45b6-9348-03864a9a7015 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:27.595323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785e13-e2c6-45b6-9348-03864a9a7015 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:27.595587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a785e13-e2c6-45b6-9348-03864a9a7015 None admin] PATH_INFO: `/v3/domains/5e13d104e2024b83a905cb9b7b11ed59` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:27.606012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a785e13-e2c6-45b6-9348-03864a9a7015 None admin] RBAC: Authorizing `identity:get_domain(domain_id=5e13d104e2024b83a905cb9b7b11ed59)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:27.607458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a785e13-e2c6-45b6-9348-03864a9a7015 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:27.609001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 457/885] 10.222.0.20 () {56 vars in 1183 bytes} [Wed Oct 19 21:44:27 2022] GET /identity/v3/domains/5e13d104e2024b83a905cb9b7b11ed59 => generated 267 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:27.614891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993c9c84-7e41-42ca-9cc5-1b5aa0caee35 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:27.709961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993c9c84-7e41-42ca-9cc5-1b5aa0caee35 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:27.745581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1628578-b2bb-414e-9eef-fdcec08942e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:27.746112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1628578-b2bb-414e-9eef-fdcec08942e5 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:27.746385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1628578-b2bb-414e-9eef-fdcec08942e5 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:27.746651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1628578-b2bb-414e-9eef-fdcec08942e5 None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:27.747330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1628578-b2bb-414e-9eef-fdcec08942e5 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:27.748754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1628578-b2bb-414e-9eef-fdcec08942e5 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:27.834986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:27.841592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 429/886] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:44:27 2022] POST /identity/v3/projects => generated 361 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:44:29.986388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7427fa75-a613-4154-b413-2da830a34aba None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:29.986927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7427fa75-a613-4154-b413-2da830a34aba None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:29.986927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7427fa75-a613-4154-b413-2da830a34aba None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:29.988279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 458/887] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:29.993451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35873f9-b911-4347-8e94-cc423468fbc4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:29.993870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35873f9-b911-4347-8e94-cc423468fbc4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:29.993999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35873f9-b911-4347-8e94-cc423468fbc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:30.012738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b35873f9-b911-4347-8e94-cc423468fbc4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:30.013909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b35873f9-b911-4347-8e94-cc423468fbc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:30.077961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 430/888] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:29 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:30.347085 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7427fa75-a613-4154-b413-2da830a34aba None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:30.412613 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:30.445062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7427fa75-a613-4154-b413-2da830a34aba None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:30.477627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57ac3e6-8018-4916-81ed-d0619c09c82a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:30.478243 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ac3e6-8018-4916-81ed-d0619c09c82a None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:30.478526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ac3e6-8018-4916-81ed-d0619c09c82a None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:30.478808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57ac3e6-8018-4916-81ed-d0619c09c82a None admin] PATH_INFO: `/v3/domains/5e13d104e2024b83a905cb9b7b11ed59` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:30.479950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57ac3e6-8018-4916-81ed-d0619c09c82a None admin] RBAC: Authorizing `identity:get_domain(domain_id=5e13d104e2024b83a905cb9b7b11ed59)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:30.481778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57ac3e6-8018-4916-81ed-d0619c09c82a None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:30.482925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 459/889] 10.222.0.20 () {56 vars in 1183 bytes} [Wed Oct 19 21:44:30 2022] GET /identity/v3/domains/5e13d104e2024b83a905cb9b7b11ed59 => generated 267 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:30.488619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35873f9-b911-4347-8e94-cc423468fbc4 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:30.584876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35873f9-b911-4347-8e94-cc423468fbc4 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:30.612375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:30.617842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf2ed81-971b-4a73-bbf1-fa14dc13029e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:30.618428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2ed81-971b-4a73-bbf1-fa14dc13029e None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:30.618700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2ed81-971b-4a73-bbf1-fa14dc13029e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:30.618972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2ed81-971b-4a73-bbf1-fa14dc13029e None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:30.619650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf2ed81-971b-4a73-bbf1-fa14dc13029e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:30.621069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf2ed81-971b-4a73-bbf1-fa14dc13029e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:30.704228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 431/890] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:44:30 2022] POST /identity/v3/users => generated 311 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:44:32.845602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f44e91-f9d5-4110-a2b0-3b3fe05d8823 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:32.846150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f44e91-f9d5-4110-a2b0-3b3fe05d8823 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:32.846150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f44e91-f9d5-4110-a2b0-3b3fe05d8823 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:32.847541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 460/891] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:32.852634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea68278-bf77-49dc-b312-3ba7b3e7577e None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:32.853086 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea68278-bf77-49dc-b312-3ba7b3e7577e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:32.853194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea68278-bf77-49dc-b312-3ba7b3e7577e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:32.871964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8ea68278-bf77-49dc-b312-3ba7b3e7577e None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:32.873145 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8ea68278-bf77-49dc-b312-3ba7b3e7577e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:32.937012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 432/892] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:33.203927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f44e91-f9d5-4110-a2b0-3b3fe05d8823 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:33.301709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f44e91-f9d5-4110-a2b0-3b3fe05d8823 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:33.334272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aa070d-f34d-48ae-bcd1-9fedbef34ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:33.334881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa070d-f34d-48ae-bcd1-9fedbef34ad4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:33.335181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa070d-f34d-48ae-bcd1-9fedbef34ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:33.335450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aa070d-f34d-48ae-bcd1-9fedbef34ad4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:33.337953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:33.345881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aa070d-f34d-48ae-bcd1-9fedbef34ad4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-18aa070d-f34d-48ae-bcd1-9fedbef34ad4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:44:33.350540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:44:33.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:44:33.356864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 461/893] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:44:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:44:33.358408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea68278-bf77-49dc-b312-3ba7b3e7577e None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:33.454601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea68278-bf77-49dc-b312-3ba7b3e7577e None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:33.470166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:33.487404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:33.487992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:33.488264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:33.488529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:33.489191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:44:33.489488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:33.491012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:33.499287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 433/894] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:44:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:44:33.505132 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aa070d-f34d-48ae-bcd1-9fedbef34ad4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:33.602488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aa070d-f34d-48ae-bcd1-9fedbef34ad4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:33.634071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f251b02-3177-4372-88ec-c979662ac483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:33.634689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f251b02-3177-4372-88ec-c979662ac483 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:33.634985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f251b02-3177-4372-88ec-c979662ac483 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:33.635791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f251b02-3177-4372-88ec-c979662ac483 None admin] PATH_INFO: `/v3/users/03cca868d8ce435388d7abe09cb7564c` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:33.649029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f251b02-3177-4372-88ec-c979662ac483 None admin] RBAC: Authorizing `identity:get_user(user_id=03cca868d8ce435388d7abe09cb7564c)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:33.650463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f251b02-3177-4372-88ec-c979662ac483 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:33.652728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 462/895] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:44:33 2022] GET /identity/v3/users/03cca868d8ce435388d7abe09cb7564c => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:33.658755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:33.753477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81362b5-e91c-45ea-8062-554b97441d31 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:33.786337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f39ac14-fedc-4af2-89ce-4719d8e793dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:33.787089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f39ac14-fedc-4af2-89ce-4719d8e793dc None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:33.787252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f39ac14-fedc-4af2-89ce-4719d8e793dc None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:33.787520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f39ac14-fedc-4af2-89ce-4719d8e793dc None admin] PATH_INFO: `/v3/projects/d8536e358d43451297fee0e1b389599c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:33.788758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f39ac14-fedc-4af2-89ce-4719d8e793dc None admin] RBAC: Authorizing `identity:get_project(project_id=d8536e358d43451297fee0e1b389599c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:33.790219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f39ac14-fedc-4af2-89ce-4719d8e793dc None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:33.791954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 434/896] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:44:33 2022] GET /identity/v3/projects/d8536e358d43451297fee0e1b389599c => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:33.797769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f251b02-3177-4372-88ec-c979662ac483 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:33.850887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:33.893698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f251b02-3177-4372-88ec-c979662ac483 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:33.925810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa4674dc-9787-4810-8e00-9bc012b9aed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:33.926449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4674dc-9787-4810-8e00-9bc012b9aed7 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:33.926721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4674dc-9787-4810-8e00-9bc012b9aed7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:33.927015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa4674dc-9787-4810-8e00-9bc012b9aed7 None admin] PATH_INFO: `/v3/projects/d8536e358d43451297fee0e1b389599c/users/03cca868d8ce435388d7abe09cb7564c/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:33.928869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4674dc-9787-4810-8e00-9bc012b9aed7 None admin] RBAC: Authorizing `identity:create_grant(project_id=d8536e358d43451297fee0e1b389599c, user_id=03cca868d8ce435388d7abe09cb7564c, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:33.930270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa4674dc-9787-4810-8e00-9bc012b9aed7 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:33.998721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 463/897] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:44:33 2022] PUT /identity/v3/projects/d8536e358d43451297fee0e1b389599c/users/03cca868d8ce435388d7abe09cb7564c/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:44:36.111671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c006245-ed62-4127-ba2c-22b97178e78e None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:36.112180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c006245-ed62-4127-ba2c-22b97178e78e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:36.112180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c006245-ed62-4127-ba2c-22b97178e78e None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:36.113464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 435/898] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:44:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:44:36.118519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34d8919-ae7d-4907-81f6-fd54d90fb49e None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:36.118973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34d8919-ae7d-4907-81f6-fd54d90fb49e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:36.119081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34d8919-ae7d-4907-81f6-fd54d90fb49e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:36.137633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c34d8919-ae7d-4907-81f6-fd54d90fb49e None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:44:36.138827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c34d8919-ae7d-4907-81f6-fd54d90fb49e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:44:36.202749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 464/899] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:44:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:44:36.474274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c006245-ed62-4127-ba2c-22b97178e78e None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:36.520287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:36.573270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c006245-ed62-4127-ba2c-22b97178e78e None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:36.606078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8553a40-fbf4-4414-8285-b054e3166a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:36.606699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8553a40-fbf4-4414-8285-b054e3166a88 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:36.606997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8553a40-fbf4-4414-8285-b054e3166a88 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:36.607270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8553a40-fbf4-4414-8285-b054e3166a88 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:36.617358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8553a40-fbf4-4414-8285-b054e3166a88 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-a8553a40-fbf4-4414-8285-b054e3166a88 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:44:36.622162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:44:36.623252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:44:36.624275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 436/900] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:44:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:44:36.629271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34d8919-ae7d-4907-81f6-fd54d90fb49e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:36.673154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:36.723588 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34d8919-ae7d-4907-81f6-fd54d90fb49e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:36.756369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:36.756868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:36.757146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:36.757414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:36.758065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:44:36.758364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:36.759746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:36.768087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 465/901] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:44:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:36.773851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8553a40-fbf4-4414-8285-b054e3166a88 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:36.870438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8553a40-fbf4-4414-8285-b054e3166a88 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:36.877508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:36.903311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201d7304-4105-4572-8146-6ae0a7b13feb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:36.903972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201d7304-4105-4572-8146-6ae0a7b13feb None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:36.904470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201d7304-4105-4572-8146-6ae0a7b13feb None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:36.905246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201d7304-4105-4572-8146-6ae0a7b13feb None admin] PATH_INFO: `/v3/users/03cca868d8ce435388d7abe09cb7564c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:36.906373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201d7304-4105-4572-8146-6ae0a7b13feb None admin] RBAC: Authorizing `identity:get_user(user_id=03cca868d8ce435388d7abe09cb7564c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:36.907840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-201d7304-4105-4572-8146-6ae0a7b13feb None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:36.909410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 437/902] 10.222.0.20 () {56 vars in 1177 bytes} [Wed Oct 19 21:44:36 2022] GET /identity/v3/users/03cca868d8ce435388d7abe09cb7564c => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:36.915377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:37.010112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd44fada-496d-4abf-9acb-0ac49acac3b6 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:37.017118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:37.042743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d43ad6f-f020-4b38-bfd4-82b15cecb221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:37.043370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d43ad6f-f020-4b38-bfd4-82b15cecb221 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:37.043648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d43ad6f-f020-4b38-bfd4-82b15cecb221 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:37.043919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d43ad6f-f020-4b38-bfd4-82b15cecb221 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:37.107561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d43ad6f-f020-4b38-bfd4-82b15cecb221 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:37.109028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d43ad6f-f020-4b38-bfd4-82b15cecb221 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:37.202924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 466/903] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:44:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:44:37.208864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201d7304-4105-4572-8146-6ae0a7b13feb None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:37.303239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201d7304-4105-4572-8146-6ae0a7b13feb None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:37.340460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0abf5b6f-cf4a-449b-a4ba-65b13d517db1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:37.341083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf5b6f-cf4a-449b-a4ba-65b13d517db1 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:37.341362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf5b6f-cf4a-449b-a4ba-65b13d517db1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:37.341633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0abf5b6f-cf4a-449b-a4ba-65b13d517db1 None admin] PATH_INFO: `/v3/projects/d8536e358d43451297fee0e1b389599c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:37.342740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abf5b6f-cf4a-449b-a4ba-65b13d517db1 None admin] RBAC: Authorizing `identity:get_project(project_id=d8536e358d43451297fee0e1b389599c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:37.344200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0abf5b6f-cf4a-449b-a4ba-65b13d517db1 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:37.345821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 438/904] 10.222.0.20 () {56 vars in 1186 bytes} [Wed Oct 19 21:44:37 2022] GET /identity/v3/projects/d8536e358d43451297fee0e1b389599c => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:37.351836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d43ad6f-f020-4b38-bfd4-82b15cecb221 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:44:37.360180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:37.448216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d43ad6f-f020-4b38-bfd4-82b15cecb221 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:44:37.481263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9af341-e192-40af-8cf9-6d6181c3b335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:44:37.481845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9af341-e192-40af-8cf9-6d6181c3b335 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:44:37.482120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9af341-e192-40af-8cf9-6d6181c3b335 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:44:37.482390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9af341-e192-40af-8cf9-6d6181c3b335 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:44:37.483130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9af341-e192-40af-8cf9-6d6181c3b335 None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=d8536e358d43451297fee0e1b389599c, user.id=03cca868d8ce435388d7abe09cb7564c) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:44:37.483439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9af341-e192-40af-8cf9-6d6181c3b335 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:44:37.484845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9af341-e192-40af-8cf9-6d6181c3b335 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:44:37.490383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 467/905] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:44:37 2022] GET /identity/v3/role_assignments?user.id=03cca868d8ce435388d7abe09cb7564c&scope.project.id=d8536e358d43451297fee0e1b389599c&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:44:47.335938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=79 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:44:47.487872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=65 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:50.042376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf56ab2-e24e-4788-9ac5-5d81b8514763 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:50.042921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf56ab2-e24e-4788-9ac5-5d81b8514763 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:50.042921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf56ab2-e24e-4788-9ac5-5d81b8514763 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:50.044285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 439/906] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:45:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:45:50.050096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99484c86-3d5b-44c4-b87a-45134c192529 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:50.054260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99484c86-3d5b-44c4-b87a-45134c192529 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:50.054260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99484c86-3d5b-44c4-b87a-45134c192529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:50.068525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-99484c86-3d5b-44c4-b87a-45134c192529 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:45:50.069714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-99484c86-3d5b-44c4-b87a-45134c192529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:45:50.134125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 468/907] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:45:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:45:50.399507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf56ab2-e24e-4788-9ac5-5d81b8514763 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:50.498181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf56ab2-e24e-4788-9ac5-5d81b8514763 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:50.530837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:50.531488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:50.531809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:50.532341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:50.542179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:50.543646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:45:50.558136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:50.559415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:45:50.560568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:45:50.560568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:50.560568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:45:50.560568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 440/908] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:45:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:50.566068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99484c86-3d5b-44c4-b87a-45134c192529 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:50.661380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99484c86-3d5b-44c4-b87a-45134c192529 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:50.689690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:50.695414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ad7044-ec68-42ec-af2d-8200f7c842b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:50.697216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ad7044-ec68-42ec-af2d-8200f7c842b1 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:50.697216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ad7044-ec68-42ec-af2d-8200f7c842b1 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:50.697216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ad7044-ec68-42ec-af2d-8200f7c842b1 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:50.697216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ad7044-ec68-42ec-af2d-8200f7c842b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:50.697600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ad7044-ec68-42ec-af2d-8200f7c842b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:50.698813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ad7044-ec68-42ec-af2d-8200f7c842b1 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:50.709583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 469/909] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:45:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:50.715358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:50.768681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:50.814849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8209ac64-7222-455d-bec4-9bd98f4f90e6 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:50.847564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b93adc-9f41-4ad2-b3ef-eaeefb6a4703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:50.848677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b93adc-9f41-4ad2-b3ef-eaeefb6a4703 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:50.848956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b93adc-9f41-4ad2-b3ef-eaeefb6a4703 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:50.849227 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b93adc-9f41-4ad2-b3ef-eaeefb6a4703 None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:50.849881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b93adc-9f41-4ad2-b3ef-eaeefb6a4703 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:50.851356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b93adc-9f41-4ad2-b3ef-eaeefb6a4703 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:50.927722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 441/910] 10.222.0.20 () {60 vars in 1132 bytes} [Wed Oct 19 21:45:50 2022] POST /identity/v3/users => generated 251 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:45:53.076274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecda801-075d-4411-838d-57137218bd1a None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:53.076781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecda801-075d-4411-838d-57137218bd1a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:53.076781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecda801-075d-4411-838d-57137218bd1a None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:53.078070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 470/911] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:45:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:45:53.083328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66874454-bc48-4e68-b93d-5d7f76679b35 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:53.083705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66874454-bc48-4e68-b93d-5d7f76679b35 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:53.083873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66874454-bc48-4e68-b93d-5d7f76679b35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:53.102262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-66874454-bc48-4e68-b93d-5d7f76679b35 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:45:53.103480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-66874454-bc48-4e68-b93d-5d7f76679b35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:45:53.168117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 442/912] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:45:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:45:53.438803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecda801-075d-4411-838d-57137218bd1a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:53.537822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecda801-075d-4411-838d-57137218bd1a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:53.569664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fba255f-68fc-4412-82c1-c9a24f8a6ada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:53.570282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fba255f-68fc-4412-82c1-c9a24f8a6ada None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:53.570585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fba255f-68fc-4412-82c1-c9a24f8a6ada None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:53.570854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fba255f-68fc-4412-82c1-c9a24f8a6ada None admin] PATH_INFO: `/v3/roles/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:53.581692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fba255f-68fc-4412-82c1-c9a24f8a6ada None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-5fba255f-68fc-4412-82c1-c9a24f8a6ada None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:45:53.586458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:53.587570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:45:53.588427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 471/913] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:45:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:53.593539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66874454-bc48-4e68-b93d-5d7f76679b35 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:53.631586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:53.688796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66874454-bc48-4e68-b93d-5d7f76679b35 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:53.723224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:53.723829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:53.724104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:53.724373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:53.725038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] RBAC: Adding query filter params (name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:53.725339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:53.726713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:53.735723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 443/914] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:45:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:53.741551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fba255f-68fc-4412-82c1-c9a24f8a6ada None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:53.792107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:53.840924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fba255f-68fc-4412-82c1-c9a24f8a6ada None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:53.872530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:53.873145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:53.873417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:53.873684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:53.884275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:53.885698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:45:53.899820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:53.900957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:45:53.901857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:45:53.901857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:53.901857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:45:53.901857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 472/915] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:45:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:53.906813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:54.000062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:54.003387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfd037-cf98-4151-a0c7-6ecab9e10e38 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:54.036374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:54.036971 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:54.037300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:54.037570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:54.038211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:54.038510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:54.039890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:54.050885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 444/916] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:45:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:54.056781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:54.154272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6943011-c69b-4400-a5fd-a4b21344395c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:54.155879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:54.193609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:54.194253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:54.194516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:54.194786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] PATH_INFO: `/v3/users/placement` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:54.201976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:54.203538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:45:54.214195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:54.215314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:45:54.216342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 473/917] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:45:54 2022] GET /identity/v3/users/placement => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:54.221330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:54.320244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f189683-79cd-4728-b167-ce6d2fafb41a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:54.352276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:54.352893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:54.353164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:54.353434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:54.354163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:54.354467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:54.355943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:54.372924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 445/918] 10.222.0.20 () {56 vars in 1143 bytes} [Wed Oct 19 21:45:54 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:54.379463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:54.475960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f620242-df07-433d-999e-d04e872f2f8c None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:54.509130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:54.509745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:54.510024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:54.510302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:54.520629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:54.522057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:45:54.536516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:54.539959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:45:54.540888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:45:54.540888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:54.540888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:45:54.540888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 474/919] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:45:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:54.543415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:54.562904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:54.640475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7df9d7-f7a2-4446-bca5-ca511a8c235d None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:54.673652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:54.674235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:54.674510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:54.674859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:54.675529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:54.675827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:54.677172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:54.689053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 446/920] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:45:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:54.694808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:54.710513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:54.792274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f007fc2-5296-41dc-bc5e-31e3fd2a9c33 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:54.827342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:54.827975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:54.828263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:54.828535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:54.835461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:54.836905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:45:54.848465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:54.849557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:45:54.850632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 475/921] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:45:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:54.855488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:54.932753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:54.953915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c99547dd-8728-4c05-a5c4-8322305f2884 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:54.985884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7e10d5-eb8a-41e4-b51c-898f65b482be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:54.986472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7e10d5-eb8a-41e4-b51c-898f65b482be None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:54.986751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7e10d5-eb8a-41e4-b51c-898f65b482be None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:54.987045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7e10d5-eb8a-41e4-b51c-898f65b482be None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:54.987732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7e10d5-eb8a-41e4-b51c-898f65b482be None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:54.988034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7e10d5-eb8a-41e4-b51c-898f65b482be None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:54.989465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7e10d5-eb8a-41e4-b51c-898f65b482be None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:55.001509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 447/922] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:45:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:55.007669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:55.080826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:55.104042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5118d797-8791-4df6-9835-a930bf28f2c3 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:55.137706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35283de9-a586-4160-8b4b-f11812c6c0b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:55.138359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35283de9-a586-4160-8b4b-f11812c6c0b6 None admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:55.138637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35283de9-a586-4160-8b4b-f11812c6c0b6 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:55.138912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35283de9-a586-4160-8b4b-f11812c6c0b6 None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/0a6d53677cb64aaba09e89eb527879cb/roles/fa322a306cb442c79d45a314b16f1b85` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:55.152721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35283de9-a586-4160-8b4b-f11812c6c0b6 None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=0a6d53677cb64aaba09e89eb527879cb, role_id=fa322a306cb442c79d45a314b16f1b85)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:55.154162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35283de9-a586-4160-8b4b-f11812c6c0b6 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:55.221178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 476/923] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:45:55 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/0a6d53677cb64aaba09e89eb527879cb/roles/fa322a306cb442c79d45a314b16f1b85 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:45:57.332520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41256819-6826-4812-bae7-cd93c4ab73ab None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:57.333106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41256819-6826-4812-bae7-cd93c4ab73ab None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:57.333106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41256819-6826-4812-bae7-cd93c4ab73ab None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:57.334409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 448/924] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:45:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:45:57.339877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85508d-d962-49c0-b3ef-531d9f29af1b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:57.344670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85508d-d962-49c0-b3ef-531d9f29af1b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:57.344670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e85508d-d962-49c0-b3ef-531d9f29af1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:57.358712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6e85508d-d962-49c0-b3ef-531d9f29af1b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:45:57.360484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6e85508d-d962-49c0-b3ef-531d9f29af1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:45:57.426289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 477/925] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:45:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:45:57.701712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41256819-6826-4812-bae7-cd93c4ab73ab None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:57.805762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41256819-6826-4812-bae7-cd93c4ab73ab None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:57.838753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a451d-2104-422e-afb3-d1850e58c4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:57.839406 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a451d-2104-422e-afb3-d1850e58c4c1 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:57.839684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a451d-2104-422e-afb3-d1850e58c4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:57.839952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149a451d-2104-422e-afb3-d1850e58c4c1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:57.847185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:57.850984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149a451d-2104-422e-afb3-d1850e58c4c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-149a451d-2104-422e-afb3-d1850e58c4c1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:45:57.855752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:57.860954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:45:57.861965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 449/926] 10.222.0.20 () {56 vars in 1102 bytes} [Wed Oct 19 21:45:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:57.863673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85508d-d962-49c0-b3ef-531d9f29af1b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:57.916634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:57.958697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e85508d-d962-49c0-b3ef-531d9f29af1b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:57.991188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:57.991785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:57.992058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:57.992338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:57.992980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] RBAC: Adding query filter params (name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:57.993279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:57.994629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:58.002708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 478/927] 10.222.0.20 () {56 vars in 1103 bytes} [Wed Oct 19 21:45:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:58.008540 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a451d-2104-422e-afb3-d1850e58c4c1 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:58.104675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149a451d-2104-422e-afb3-d1850e58c4c1 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:58.139773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:58.140398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:58.140674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:58.140942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:58.150704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:58.152193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:45:58.166451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:58.167597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:45:58.168477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:45:58.168477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:58.168477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:45:58.168477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 450/928] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:45:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:58.173823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:58.275701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ea68be-7499-406a-9291-65e533e81600 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:58.291725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:58.309128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:58.309731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:58.310011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:58.310284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:58.310925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:58.311254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:58.312618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:58.323750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 479/929] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:45:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:58.329692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:58.378675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:58.426657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56350eae-57e0-4a0b-8883-b492b05ee2ac None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:58.457974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:58.458586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:58.458864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:58.459165 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] PATH_INFO: `/v3/users/placement` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:58.466398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:58.467820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:45:58.479073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:58.480131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:45:58.481049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 451/930] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:45:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:45:58.486652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:58.582660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15192b5f-a308-4aa4-a9f6-d509fcdcb8ef None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:58.616089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:58.616684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:58.616959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:58.617229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:58.617944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:58.618245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:58.619710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:58.636250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 480/931] 10.222.0.20 () {56 vars in 1143 bytes} [Wed Oct 19 21:45:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:58.642692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:58.739930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b588499b-762e-4e63-ba14-e20ca2606639 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:58.749648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:58.772449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7bba53-6d2c-40c6-bfef-e1a46b37ee27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:58.773039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7bba53-6d2c-40c6-bfef-e1a46b37ee27 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:58.773313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7bba53-6d2c-40c6-bfef-e1a46b37ee27 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:58.773579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7bba53-6d2c-40c6-bfef-e1a46b37ee27 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:58.839644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7bba53-6d2c-40c6-bfef-e1a46b37ee27 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:58.841108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7bba53-6d2c-40c6-bfef-e1a46b37ee27 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:58.935970 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 452/932] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:45:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 19 21:45:58.942655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:58.975179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:59.037688 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2ec68a-48ca-4241-9154-1dc132bad17d None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:59.073730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:59.074356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:59.074637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:59.074910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:59.084772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:59.086211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:45:59.101601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:59.102647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:45:59.103795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:45:59.103795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:45:59.103795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:45:59.107106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 481/933] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:45:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:45:59.108477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7bba53-6d2c-40c6-bfef-e1a46b37ee27 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:59.120467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:59.205226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7bba53-6d2c-40c6-bfef-e1a46b37ee27 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:59.238795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:59.239414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:59.239691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:59.239961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:59.240602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:59.240901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:59.242260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:59.253844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 453/934] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:45:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:45:59.259784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:59.341809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:59.356866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991513eb-5fee-42d8-860e-a117775644b7 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:59.389345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:59.389966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:59.390295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:59.390632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] PATH_INFO: `/v3/projects/service` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:59.397534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:59.399073 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:45:59.411356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:45:59.413911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:45:59.415005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 482/935] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:45:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:45:59.418542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:59.487943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:45:59.517978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746d82ef-524c-4b33-a191-222d22e51dc2 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:59.550415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948e6e04-f58d-453b-99eb-827da8a69f4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:59.551039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e6e04-f58d-453b-99eb-827da8a69f4b None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:59.551319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e6e04-f58d-453b-99eb-827da8a69f4b None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:59.551622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e6e04-f58d-453b-99eb-827da8a69f4b None admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:59.552311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948e6e04-f58d-453b-99eb-827da8a69f4b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:59.552615 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948e6e04-f58d-453b-99eb-827da8a69f4b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:59.554043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-948e6e04-f58d-453b-99eb-827da8a69f4b None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:59.566061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 454/936] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:45:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:45:59.572135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:45:59.671798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71515b85-a653-4ecb-a764-5621dbf96ced None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:45:59.705896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81f1890-f497-4643-aa29-0dd9d67c8c43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:45:59.706539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f1890-f497-4643-aa29-0dd9d67c8c43 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:45:59.706813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f1890-f497-4643-aa29-0dd9d67c8c43 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:45:59.707877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81f1890-f497-4643-aa29-0dd9d67c8c43 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:45:59.707877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81f1890-f497-4643-aa29-0dd9d67c8c43 None admin] RBAC: Adding query filter params (role.id=fa322a306cb442c79d45a314b16f1b85, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=0a6d53677cb64aaba09e89eb527879cb) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:45:59.708105 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81f1890-f497-4643-aa29-0dd9d67c8c43 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:45:59.709507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81f1890-f497-4643-aa29-0dd9d67c8c43 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:45:59.715062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 483/937] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:45:59 2022] GET /identity/v3/role_assignments?user.id=0a6d53677cb64aaba09e89eb527879cb&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=fa322a306cb442c79d45a314b16f1b85 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:01.880906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487be71d-21d7-422c-8847-6404becd2e34 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:01.881434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487be71d-21d7-422c-8847-6404becd2e34 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:01.881434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487be71d-21d7-422c-8847-6404becd2e34 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:01.882825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 455/938] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:46:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:46:01.888048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68d3139-1581-4811-8d4c-8cf528022403 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:01.888447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68d3139-1581-4811-8d4c-8cf528022403 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:01.888663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68d3139-1581-4811-8d4c-8cf528022403 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:01.906344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f68d3139-1581-4811-8d4c-8cf528022403 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:46:01.907876 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f68d3139-1581-4811-8d4c-8cf528022403 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:46:01.912613 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:01.972208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 484/939] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:46:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:46:02.237695 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487be71d-21d7-422c-8847-6404becd2e34 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:02.335790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487be71d-21d7-422c-8847-6404becd2e34 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:02.364945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:02.370645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b075fac-58e0-4a47-85d5-e74df0bfc100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:02.371300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b075fac-58e0-4a47-85d5-e74df0bfc100 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:02.371574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b075fac-58e0-4a47-85d5-e74df0bfc100 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:02.371842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b075fac-58e0-4a47-85d5-e74df0bfc100 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:02.381445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b075fac-58e0-4a47-85d5-e74df0bfc100 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-5b075fac-58e0-4a47-85d5-e74df0bfc100 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:46:02.386156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:02.391794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:46:02.392892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 456/940] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:46:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:02.394081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68d3139-1581-4811-8d4c-8cf528022403 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:02.491709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68d3139-1581-4811-8d4c-8cf528022403 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:02.524976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:02.525579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:02.525858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:02.526132 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:02.526790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] RBAC: Adding query filter params (name=admin) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:02.527126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:02.528493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:02.536616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 485/941] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:46:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:02.542446 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b075fac-58e0-4a47-85d5-e74df0bfc100 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:02.640266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b075fac-58e0-4a47-85d5-e74df0bfc100 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:02.674043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:02.674699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:02.675012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:02.675281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:02.684933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:02.686364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:46:02.700944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:02.702028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:46:02.703271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:46:02.703271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:46:02.703271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:46:02.703271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 457/942] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:46:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:02.707905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:02.710729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:02.807482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cbc8cdb-f421-4465-a5b0-52d4ea06ca48 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:02.839232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:02.839822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:02.840104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:02.840372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:02.841004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:02.841299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:02.842643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:02.855000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 486/943] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:46:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:02.861295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:02.902920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:02.957923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60166d1-2f17-4360-9e71-6b8c381f505f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:02.992076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:02.992691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:02.992960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:02.993229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] PATH_INFO: `/v3/users/placement` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:03.000031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:03.001526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:46:03.013499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:03.014555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:46:03.015594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 458/944] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:46:02 2022] GET /identity/v3/users/placement => generated 87 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:03.020437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:03.080394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:03.117200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4795f29a-a5c8-43bb-bcc8-dc3d0a6ed994 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:03.149206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:03.149854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:03.150127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:03.150396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:03.151141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:03.151441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:03.152860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:03.169918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 487/945] 10.222.0.20 () {56 vars in 1143 bytes} [Wed Oct 19 21:46:03 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:03.175913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:03.273475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa06893-6462-48eb-9cea-c69956b30f33 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:03.274968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:03.306701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:03.307352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:03.307634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:03.307908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:03.317440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:03.319518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:46:03.335056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:03.336077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:46:03.337379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:46:03.337379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:46:03.337379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:46:03.337379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 459/946] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:46:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:03.341963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:03.439529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee4c84-1953-41de-aa5e-3aac351914df None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:03.459053 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:03.472434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:03.473029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:03.473305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:03.473575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:03.474207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:03.474609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:03.476000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:03.487489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 488/947] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:46:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:03.493309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:03.591017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8546ffe8-e3d5-49a7-9398-6e4e071dcf3f None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:03.624312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:03.624930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:03.625208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:03.625499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:03.631904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:03.633341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:46:03.644327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:03.645374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:46:03.646496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 460/948] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:46:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:03.651425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:03.749469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f179da0f-b806-4e6d-a79a-1b3b8974789b None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:03.785165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db95f59-ce8c-4d6f-b06e-8a2d4eed906b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:03.786242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db95f59-ce8c-4d6f-b06e-8a2d4eed906b None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:03.786530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db95f59-ce8c-4d6f-b06e-8a2d4eed906b None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:03.786810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db95f59-ce8c-4d6f-b06e-8a2d4eed906b None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:03.788680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db95f59-ce8c-4d6f-b06e-8a2d4eed906b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:03.789002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db95f59-ce8c-4d6f-b06e-8a2d4eed906b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:03.790465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db95f59-ce8c-4d6f-b06e-8a2d4eed906b None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:03.803646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 489/949] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:46:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 21:46:03.809814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:03.827415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:03.909096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403a932a-6620-42bb-9e80-758b2539d1b3 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:03.943765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-964ac360-af54-4373-9aff-65b285fd9ed1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:03.944424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ac360-af54-4373-9aff-65b285fd9ed1 None admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:03.944707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ac360-af54-4373-9aff-65b285fd9ed1 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:03.945172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-964ac360-af54-4373-9aff-65b285fd9ed1 None admin] PATH_INFO: `/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/0a6d53677cb64aaba09e89eb527879cb/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:03.947155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964ac360-af54-4373-9aff-65b285fd9ed1 None admin] RBAC: Authorizing `identity:create_grant(project_id=efdf8a1e8219440c8d2a0e7395e2b1f4, user_id=0a6d53677cb64aaba09e89eb527879cb, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:03.948592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-964ac360-af54-4373-9aff-65b285fd9ed1 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:04.036570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 461/950] 10.222.0.20 () {58 vars in 1439 bytes} [Wed Oct 19 21:46:03 2022] PUT /identity/v3/projects/efdf8a1e8219440c8d2a0e7395e2b1f4/users/0a6d53677cb64aaba09e89eb527879cb/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:46:06.143675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a67176-f1f2-4656-82a5-0a3ccd486862 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:06.144224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a67176-f1f2-4656-82a5-0a3ccd486862 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:06.144224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a67176-f1f2-4656-82a5-0a3ccd486862 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:06.145484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 490/951] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:46:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:46:06.150827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec87906-2a35-439c-81ba-a3110449357e None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:06.151394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec87906-2a35-439c-81ba-a3110449357e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:06.151394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec87906-2a35-439c-81ba-a3110449357e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:06.169895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8ec87906-2a35-439c-81ba-a3110449357e None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:46:06.171110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8ec87906-2a35-439c-81ba-a3110449357e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:46:06.237726 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 462/952] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:46:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:46:06.520976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a67176-f1f2-4656-82a5-0a3ccd486862 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:06.566510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:06.625786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a67176-f1f2-4656-82a5-0a3ccd486862 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:06.663322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a3f92-80d1-4109-a59d-ae921dbf7129 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:06.663937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a3f92-80d1-4109-a59d-ae921dbf7129 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:06.664209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a3f92-80d1-4109-a59d-ae921dbf7129 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:06.664822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a3f92-80d1-4109-a59d-ae921dbf7129 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:06.674858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273a3f92-80d1-4109-a59d-ae921dbf7129 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-273a3f92-80d1-4109-a59d-ae921dbf7129 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 19 21:46:06.679694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:06.680894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:06.682008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 491/953] 10.222.0.20 () {56 vars in 1096 bytes} [Wed Oct 19 21:46:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:06.686978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec87906-2a35-439c-81ba-a3110449357e None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:06.690122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:06.800023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec87906-2a35-439c-81ba-a3110449357e None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:06.851431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:06.852064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:06.857280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:06.857560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:06.858245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] RBAC: Adding query filter params (name=admin) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:06.858552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:06.860677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:06.868867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 463/954] 10.222.0.20 () {56 vars in 1099 bytes} [Wed Oct 19 21:46:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:06.874788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a3f92-80d1-4109-a59d-ae921dbf7129 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:06.971606 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:06.974598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a3f92-80d1-4109-a59d-ae921dbf7129 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:07.007759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:07.008376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:07.008651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:07.008916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:07.018475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:07.024300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:46:07.033960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:07.035152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:46:07.036421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:46:07.036421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:46:07.036421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:07.036421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 492/955] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:46:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:07.041587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:07.106018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:07.141031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ada10d-530a-440b-b417-94e2cb0d7b5a None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:07.174007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:07.174646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:07.174866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:07.175183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:07.175810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:07.176112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:07.177481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:07.189431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 464/956] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:46:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:07.195178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:07.291723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0d71a-2410-42f0-b6aa-b9708fbd3182 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:07.324078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:07.324690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:07.324964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:07.325231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] PATH_INFO: `/v3/users/placement` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:07.332294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:07.333715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 19 21:46:07.346061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:07.353672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:07.354839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 493/957] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:46:07 2022] GET /identity/v3/users/placement => generated 87 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:07.355681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:07.449271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3786d35-992c-481b-9d53-eddb2dbb9de7 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:07.473961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:07.482010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:07.482620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:07.482884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:07.483185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:07.483911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:07.484215 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] RBAC: Authorizing `identity:list_users()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:07.486184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:07.502371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 465/958] 10.222.0.20 () {56 vars in 1143 bytes} [Wed Oct 19 21:46:07 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:07.508195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:07.523053 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:07.605833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1dbecc6-5fb1-4183-976e-fa37318eb818 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:07.638090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbab682-0f8e-4a6d-b14e-b83c2efe87cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:07.638684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbab682-0f8e-4a6d-b14e-b83c2efe87cc None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:07.638970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbab682-0f8e-4a6d-b14e-b83c2efe87cc None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:07.639253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbab682-0f8e-4a6d-b14e-b83c2efe87cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:07.704886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbab682-0f8e-4a6d-b14e-b83c2efe87cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:07.706316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbab682-0f8e-4a6d-b14e-b83c2efe87cc None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:07.719086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:07.808040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 494/959] 10.222.0.20 () {58 vars in 1282 bytes} [Wed Oct 19 21:46:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 19 21:46:07.814620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:07.914441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d261bd8e-a7cf-4b40-9b37-0606ecbb6dde None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:07.948973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:07.949601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:07.949881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:07.950154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:07.959973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:07.961552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 19 21:46:07.976655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:07.978365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 19 21:46:07.979415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 19 21:46:07.979415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 19 21:46:07.979415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:46:07.979415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 466/960] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:46:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 19 21:46:07.983690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbab682-0f8e-4a6d-b14e-b83c2efe87cc None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:08.079871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbab682-0f8e-4a6d-b14e-b83c2efe87cc None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:08.093376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:08.114595 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:08.115222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:08.115504 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:08.115778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:08.116416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:08.116720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:08.118086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:08.129002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 495/961] 10.222.0.20 () {56 vars in 1109 bytes} [Wed Oct 19 21:46:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:08.134874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:08.170583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:08.232804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcfb248d-b99c-45c2-b500-30a092e99f70 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:08.266860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:08.267737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:08.268183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:08.268500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] PATH_INFO: `/v3/projects/service` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:08.275548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:08.277028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 19 21:46:08.288901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:08.296177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 21:46:08.297273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 467/962] 10.222.0.20 () {56 vars in 1111 bytes} [Wed Oct 19 21:46:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:08.298471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:08.398231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9ed992-47df-4741-97e6-2490a3c96f22 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:08.431366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dcbacc-6dea-4be2-b269-15b028e55685 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:08.431925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dcbacc-6dea-4be2-b269-15b028e55685 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:08.432206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dcbacc-6dea-4be2-b269-15b028e55685 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:08.432476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dcbacc-6dea-4be2-b269-15b028e55685 None admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:08.433156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dcbacc-6dea-4be2-b269-15b028e55685 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:08.433456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dcbacc-6dea-4be2-b269-15b028e55685 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:08.434873 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dcbacc-6dea-4be2-b269-15b028e55685 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:08.447013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 496/963] 10.222.0.20 () {56 vars in 1148 bytes} [Wed Oct 19 21:46:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:08.453190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:08.543660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:08.549749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9beb098-09dc-4ec1-bbbe-a8eb945dae4c None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:08.582140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755d8f8e-25ac-4631-ab36-dadbe2a73fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:08.582722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755d8f8e-25ac-4631-ab36-dadbe2a73fe5 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:08.583021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755d8f8e-25ac-4631-ab36-dadbe2a73fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:08.583291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755d8f8e-25ac-4631-ab36-dadbe2a73fe5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:08.584002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755d8f8e-25ac-4631-ab36-dadbe2a73fe5 None admin] RBAC: Adding query filter params (role.id=5573fc131a7b421585c72c4a56d84f4b, scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4, user.id=0a6d53677cb64aaba09e89eb527879cb) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:08.584305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755d8f8e-25ac-4631-ab36-dadbe2a73fe5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:08.585715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755d8f8e-25ac-4631-ab36-dadbe2a73fe5 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:08.591155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 468/964] 10.222.0.20 () {56 vars in 1374 bytes} [Wed Oct 19 21:46:08 2022] GET /identity/v3/role_assignments?user.id=0a6d53677cb64aaba09e89eb527879cb&scope.project.id=efdf8a1e8219440c8d2a0e7395e2b1f4&role.id=5573fc131a7b421585c72c4a56d84f4b => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:10.737714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562db2cd-4bc8-442e-8ba1-16332d777f38 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:10.738286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562db2cd-4bc8-442e-8ba1-16332d777f38 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:10.738286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562db2cd-4bc8-442e-8ba1-16332d777f38 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:10.739624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 497/965] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:46:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:46:10.744812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa2eeb-ba8f-41da-a290-d96428631c89 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:10.745226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa2eeb-ba8f-41da-a290-d96428631c89 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:10.745389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fa2eeb-ba8f-41da-a290-d96428631c89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:10.763201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a3fa2eeb-ba8f-41da-a290-d96428631c89 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:46:10.764373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a3fa2eeb-ba8f-41da-a290-d96428631c89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:46:10.830023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 469/966] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:46:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:46:11.099922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562db2cd-4bc8-442e-8ba1-16332d777f38 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:11.130338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:11.200096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562db2cd-4bc8-442e-8ba1-16332d777f38 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:11.232859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:11.233468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:11.233741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:11.234009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] PATH_INFO: `/v3/services/placement` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:11.234577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:11.236572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:11.245017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:11.246222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:11.247252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 498/967] 10.222.0.20 () {56 vars in 1117 bytes} [Wed Oct 19 21:46:11 2022] GET /identity/v3/services/placement => generated 90 bytes in 147 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:11.252159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa2eeb-ba8f-41da-a290-d96428631c89 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:11.338574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:11.348483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fa2eeb-ba8f-41da-a290-d96428631c89 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:11.382707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b45554d-1b9d-421a-8d90-88271012afd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:11.383273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b45554d-1b9d-421a-8d90-88271012afd8 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:11.383556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b45554d-1b9d-421a-8d90-88271012afd8 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:11.383833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b45554d-1b9d-421a-8d90-88271012afd8 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:11.384472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b45554d-1b9d-421a-8d90-88271012afd8 None admin] RBAC: Adding query filter params (name=placement) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:11.384843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b45554d-1b9d-421a-8d90-88271012afd8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:11.386273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b45554d-1b9d-421a-8d90-88271012afd8 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:11.391647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 470/968] 10.222.0.20 () {56 vars in 1116 bytes} [Wed Oct 19 21:46:11 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:11.397453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:11.490977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:11.496565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0949d33-d485-407e-a2dc-02a2ca153881 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:11.528738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e04eeca-76a4-4f11-b0e0-9e71a6fd8708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:11.529336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04eeca-76a4-4f11-b0e0-9e71a6fd8708 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:11.529612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04eeca-76a4-4f11-b0e0-9e71a6fd8708 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:11.529889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04eeca-76a4-4f11-b0e0-9e71a6fd8708 None admin] PATH_INFO: `/v3/services` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:11.530517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e04eeca-76a4-4f11-b0e0-9e71a6fd8708 None admin] RBAC: Adding query filter params (type=placement) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:11.530820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e04eeca-76a4-4f11-b0e0-9e71a6fd8708 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:11.533049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e04eeca-76a4-4f11-b0e0-9e71a6fd8708 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:11.537293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 499/969] 10.222.0.20 () {56 vars in 1116 bytes} [Wed Oct 19 21:46:11 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:13.669836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9838af1e-3e88-4d02-b0f5-7ab6d5ecafd6 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:13.670402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9838af1e-3e88-4d02-b0f5-7ab6d5ecafd6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:13.670402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9838af1e-3e88-4d02-b0f5-7ab6d5ecafd6 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:13.671870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 471/970] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:46:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:46:13.677164 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06ee10-3d22-4efd-af3c-6c030adf464c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:13.677571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06ee10-3d22-4efd-af3c-6c030adf464c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:13.677796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06ee10-3d22-4efd-af3c-6c030adf464c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:13.695327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-1e06ee10-3d22-4efd-af3c-6c030adf464c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:46:13.696498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1e06ee10-3d22-4efd-af3c-6c030adf464c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:46:13.760532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 500/971] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:46:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 19 21:46:14.042022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9838af1e-3e88-4d02-b0f5-7ab6d5ecafd6 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:14.143035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9838af1e-3e88-4d02-b0f5-7ab6d5ecafd6 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:14.177180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057415ab-9945-4a0e-9e26-4724f42634f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:14.177776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057415ab-9945-4a0e-9e26-4724f42634f6 None admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:14.178057 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057415ab-9945-4a0e-9e26-4724f42634f6 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:14.178329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057415ab-9945-4a0e-9e26-4724f42634f6 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:14.179018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057415ab-9945-4a0e-9e26-4724f42634f6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:14.180423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057415ab-9945-4a0e-9e26-4724f42634f6 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:14.258791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 472/972] 10.222.0.20 () {60 vars in 1141 bytes} [Wed Oct 19 21:46:14 2022] POST /identity/v3/services => generated 246 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:46:16.417785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7721808-5a63-43c2-9ef7-8d88739fc0cb None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:16.418351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7721808-5a63-43c2-9ef7-8d88739fc0cb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:16.418351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7721808-5a63-43c2-9ef7-8d88739fc0cb None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:16.419734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 501/973] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:46:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:46:16.425486 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef52d9-8364-4596-ab72-24628eda592f None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:16.425949 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef52d9-8364-4596-ab72-24628eda592f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:16.426063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aef52d9-8364-4596-ab72-24628eda592f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:16.444033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5aef52d9-8364-4596-ab72-24628eda592f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:46:16.447975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5aef52d9-8364-4596-ab72-24628eda592f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:46:16.454090 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:16.515365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 473/974] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:46:16 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:46:16.783289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7721808-5a63-43c2-9ef7-8d88739fc0cb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:16.870512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:16.880927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7721808-5a63-43c2-9ef7-8d88739fc0cb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:16.912997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:16.913612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:16.915798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:16.915798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] PATH_INFO: `/v3/services/placement` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:16.915798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:16.916202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:16.925138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:16.926252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:16.927119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 502/975] 10.222.0.20 () {56 vars in 1117 bytes} [Wed Oct 19 21:46:16 2022] GET /identity/v3/services/placement => generated 90 bytes in 144 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:16.932515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aef52d9-8364-4596-ab72-24628eda592f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:17.027937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aef52d9-8364-4596-ab72-24628eda592f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:17.061311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:17.061838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:17.062113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:17.062474 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:17.063215 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] RBAC: Adding query filter params (name=placement) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:17.063627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:17.064920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:17.069882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 474/976] 10.222.0.20 () {56 vars in 1116 bytes} [Wed Oct 19 21:46:16 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:17.075798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:17.174822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f92f3009-9a35-4dc2-b311-aaac168741c4 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:17.206689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad28e2e5-7d65-48c6-aa87-02b3e8e618dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:17.207309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28e2e5-7d65-48c6-aa87-02b3e8e618dd None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:17.207586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28e2e5-7d65-48c6-aa87-02b3e8e618dd None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:17.207855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad28e2e5-7d65-48c6-aa87-02b3e8e618dd None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:17.208545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad28e2e5-7d65-48c6-aa87-02b3e8e618dd None admin] RBAC: Adding query filter params (interface=public, service_id=66fd85bd22b24b6093254dd016a04686, region_id=RegionOne) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:17.208850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad28e2e5-7d65-48c6-aa87-02b3e8e618dd None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:17.210198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad28e2e5-7d65-48c6-aa87-02b3e8e618dd None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:17.215027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 503/977] 10.222.0.20 () {56 vars in 1251 bytes} [Wed Oct 19 21:46:17 2022] GET /identity/v3/endpoints?service_id=66fd85bd22b24b6093254dd016a04686&interface=public®ion_id=RegionOne => generated 194 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:17.221189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:17.224882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:17.319137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4755a234-04f5-4834-9d8a-2d336745fe02 None admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:17.351897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d66295-543a-452a-abed-90d0b87ef800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:17.352496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d66295-543a-452a-abed-90d0b87ef800 None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:17.352770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d66295-543a-452a-abed-90d0b87ef800 None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:17.353040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d66295-543a-452a-abed-90d0b87ef800 None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:17.353638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d66295-543a-452a-abed-90d0b87ef800 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:17.355050 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d66295-543a-452a-abed-90d0b87ef800 None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:17.362445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 475/978] 10.222.0.20 () {56 vars in 1087 bytes} [Wed Oct 19 21:46:17 2022] GET /identity/v3/services => generated 2191 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 21:46:19.487232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f65e086-3904-4403-8573-54096932d697 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:19.487811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f65e086-3904-4403-8573-54096932d697 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:19.487811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f65e086-3904-4403-8573-54096932d697 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:19.489124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 504/979] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:46:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:46:19.494528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c9cbe-01d8-4e41-af44-96040f943dce None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:19.494972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c9cbe-01d8-4e41-af44-96040f943dce None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:19.495100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4c9cbe-01d8-4e41-af44-96040f943dce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:19.513493 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-9a4c9cbe-01d8-4e41-af44-96040f943dce None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:46:19.514664 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9a4c9cbe-01d8-4e41-af44-96040f943dce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:46:19.573693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:19.579426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 476/980] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:46:19 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 19 21:46:19.851165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f65e086-3904-4403-8573-54096932d697 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:19.870975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:19.956770 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f65e086-3904-4403-8573-54096932d697 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:19.988684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:19.989299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:19.989577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:19.989847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] PATH_INFO: `/v3/services/placement` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:19.990422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:19.991810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 21:46:20.000845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 21:46:20.002045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 21:46:20.003323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 505/981] 10.222.0.20 () {56 vars in 1117 bytes} [Wed Oct 19 21:46:19 2022] GET /identity/v3/services/placement => generated 90 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 19 21:46:20.008054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4c9cbe-01d8-4e41-af44-96040f943dce None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:20.103663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4c9cbe-01d8-4e41-af44-96040f943dce None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:20.136878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79cca680-51a4-4b6a-a92a-58135bc3ed5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:20.137465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cca680-51a4-4b6a-a92a-58135bc3ed5e None admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:20.137741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cca680-51a4-4b6a-a92a-58135bc3ed5e None admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:20.138010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cca680-51a4-4b6a-a92a-58135bc3ed5e None admin] PATH_INFO: `/v3/services` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:20.138633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cca680-51a4-4b6a-a92a-58135bc3ed5e None admin] RBAC: Adding query filter params (name=placement) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:46:20.138932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cca680-51a4-4b6a-a92a-58135bc3ed5e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:20.140319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79cca680-51a4-4b6a-a92a-58135bc3ed5e None admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:20.146095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 477/982] 10.222.0.20 () {56 vars in 1116 bytes} [Wed Oct 19 21:46:20 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:46:20.151784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:46:20.230751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:20.248008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-786deb89-621c-4a42-ac0b-b6881bb2edc2 None admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:46:20.281419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be6a604-93ca-452f-8eed-f09c0c958c13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:46:20.282007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be6a604-93ca-452f-8eed-f09c0c958c13 None admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:46:20.282282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be6a604-93ca-452f-8eed-f09c0c958c13 None admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:46:20.282549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be6a604-93ca-452f-8eed-f09c0c958c13 None admin] PATH_INFO: `/v3/endpoints` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:46:20.283498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be6a604-93ca-452f-8eed-f09c0c958c13 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:46:20.284845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be6a604-93ca-452f-8eed-f09c0c958c13 None admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:46:20.361585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 506/983] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:46:20 2022] POST /identity/v3/endpoints => generated 331 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:46:30.143025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=69 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:46:30.357123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=30,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:17.030278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b224b3-fe72-4fdc-b33e-f9cdf90111e9 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:17.030278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b224b3-fe72-4fdc-b33e-f9cdf90111e9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:17.030278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b224b3-fe72-4fdc-b33e-f9cdf90111e9 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:17.033741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fc1194-53ac-4d71-ac3b-e883dd7ffc25 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:17.034199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fc1194-53ac-4d71-ac3b-e883dd7ffc25 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:17.034199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57fc1194-53ac-4d71-ac3b-e883dd7ffc25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:17.039482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 478/984] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:17 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:17.120115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-57fc1194-53ac-4d71-ac3b-e883dd7ffc25 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:17.121376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-57fc1194-53ac-4d71-ac3b-e883dd7ffc25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:17.251226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 507/985] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:52:17.543821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea777b4-b7d9-46cf-8d4a-0c3b7ab86b66 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:17.544454 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea777b4-b7d9-46cf-8d4a-0c3b7ab86b66 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:17.544958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea777b4-b7d9-46cf-8d4a-0c3b7ab86b66 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:17.546807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 479/986] 10.222.0.20 () {54 vars in 964 bytes} [Wed Oct 19 21:52:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:17.553886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dfeb08-fd28-4774-8a29-be6a054cce69 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:17.554171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dfeb08-fd28-4774-8a29-be6a054cce69 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:17.554472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dfeb08-fd28-4774-8a29-be6a054cce69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:17.645620 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-37dfeb08-fd28-4774-8a29-be6a054cce69 None None] MFA Rules not processed for user `2f705dc9d2e541f1821dda4b5c7b7709`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:17.647012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-37dfeb08-fd28-4774-8a29-be6a054cce69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:17.671569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:17.740382 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 508/987] 10.222.0.20 () {58 vars in 1061 bytes} [Wed Oct 19 21:52:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:17.750160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea777b4-b7d9-46cf-8d4a-0c3b7ab86b66 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:17.753597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-aea777b4-b7d9-46cf-8d4a-0c3b7ab86b66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:17.893195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea777b4-b7d9-46cf-8d4a-0c3b7ab86b66 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:17.988642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427f0424-1f21-4bc1-b48d-047a9d117e38 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:17.989634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427f0424-1f21-4bc1-b48d-047a9d117e38 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:17.990327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427f0424-1f21-4bc1-b48d-047a9d117e38 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:17.990942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427f0424-1f21-4bc1-b48d-047a9d117e38 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:17.992936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-427f0424-1f21-4bc1-b48d-047a9d117e38 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:18.057854 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:18.121701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427f0424-1f21-4bc1-b48d-047a9d117e38 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:18.123902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427f0424-1f21-4bc1-b48d-047a9d117e38 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:18.148268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 480/988] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:20.714669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c27812-efd5-4455-a189-e760d01b93fb None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:20.715208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c27812-efd5-4455-a189-e760d01b93fb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:20.715298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c27812-efd5-4455-a189-e760d01b93fb None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:20.716574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 509/989] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:20.722242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485960c-07d0-43cb-9cc3-71c8b9a253b4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:20.722711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485960c-07d0-43cb-9cc3-71c8b9a253b4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:20.722813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a485960c-07d0-43cb-9cc3-71c8b9a253b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:20.792256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a485960c-07d0-43cb-9cc3-71c8b9a253b4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:20.793460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a485960c-07d0-43cb-9cc3-71c8b9a253b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:20.888113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 481/990] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:21.171288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c27812-efd5-4455-a189-e760d01b93fb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:21.173393 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-78c27812-efd5-4455-a189-e760d01b93fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:21.274446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c27812-efd5-4455-a189-e760d01b93fb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:21.343833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:21.346111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f90a1b-f333-44ea-801b-e1001830736a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:21.346709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f90a1b-f333-44ea-801b-e1001830736a service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:21.347012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f90a1b-f333-44ea-801b-e1001830736a service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:21.347287 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f90a1b-f333-44ea-801b-e1001830736a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:21.348857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f4f90a1b-f333-44ea-801b-e1001830736a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:21.454350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f90a1b-f333-44ea-801b-e1001830736a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:21.455800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f90a1b-f333-44ea-801b-e1001830736a service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:21.477872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 510/991] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:24.083515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedfd8e2-8791-405f-9263-ed1705614aad None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:24.084033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedfd8e2-8791-405f-9263-ed1705614aad None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:24.084033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eedfd8e2-8791-405f-9263-ed1705614aad None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:24.085344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 482/992] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:24.090743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c407f526-b21d-44ad-977c-ffdd3c016780 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:24.091173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c407f526-b21d-44ad-977c-ffdd3c016780 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:24.091319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c407f526-b21d-44ad-977c-ffdd3c016780 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:24.155906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c407f526-b21d-44ad-977c-ffdd3c016780 None None] MFA Rules not processed for user `7a93405c40b34cf8989eb8d1076eb475`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:24.157091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c407f526-b21d-44ad-977c-ffdd3c016780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:24.206971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:24.249118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 511/993] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:24 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:52:24.545314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedfd8e2-8791-405f-9263-ed1705614aad None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:24.546833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-eedfd8e2-8791-405f-9263-ed1705614aad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:24.557214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:24.648595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eedfd8e2-8791-405f-9263-ed1705614aad None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:24.718862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b7370f-0e38-4626-be88-97ebc952240d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:24.719497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7370f-0e38-4626-be88-97ebc952240d service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:24.719773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7370f-0e38-4626-be88-97ebc952240d service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:24.720036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7370f-0e38-4626-be88-97ebc952240d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:24.721600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-01b7370f-0e38-4626-be88-97ebc952240d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:24.781981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:24.828118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b7370f-0e38-4626-be88-97ebc952240d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:24.830152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b7370f-0e38-4626-be88-97ebc952240d service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:24.848097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 483/994] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:24 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:27.759601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08961c1-0209-4936-a769-60e941e0226f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:27.760176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08961c1-0209-4936-a769-60e941e0226f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:27.760176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c08961c1-0209-4936-a769-60e941e0226f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:27.761496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 512/995] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:27.767363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adac09-28b3-4ca8-877f-d0c1891cb196 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:27.767793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adac09-28b3-4ca8-877f-d0c1891cb196 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:27.767908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85adac09-28b3-4ca8-877f-d0c1891cb196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:27.833046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-85adac09-28b3-4ca8-877f-d0c1891cb196 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:27.834254 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-85adac09-28b3-4ca8-877f-d0c1891cb196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:27.928253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 484/996] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:27 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:52:28.214543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08961c1-0209-4936-a769-60e941e0226f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:28.216124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c08961c1-0209-4936-a769-60e941e0226f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:28.317693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c08961c1-0209-4936-a769-60e941e0226f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:28.388936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f1b8d2-16cf-4e84-8046-7c87bd699351 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:28.389529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1b8d2-16cf-4e84-8046-7c87bd699351 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:28.389802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1b8d2-16cf-4e84-8046-7c87bd699351 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:28.390067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f1b8d2-16cf-4e84-8046-7c87bd699351 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:28.391660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a8f1b8d2-16cf-4e84-8046-7c87bd699351 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:28.398355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:28.500003 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f1b8d2-16cf-4e84-8046-7c87bd699351 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:28.501351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f1b8d2-16cf-4e84-8046-7c87bd699351 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:28.518825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 513/997] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:28 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:31.701286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d6e4a4-ae90-4842-862d-8482b3cb2463 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:31.701824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d6e4a4-ae90-4842-862d-8482b3cb2463 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:31.701945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d6e4a4-ae90-4842-862d-8482b3cb2463 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:31.703891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 485/998] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:31.708811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51c91d6-a499-4dd3-b2e5-f7445b1cd09b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:31.709197 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51c91d6-a499-4dd3-b2e5-f7445b1cd09b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:31.709357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51c91d6-a499-4dd3-b2e5-f7445b1cd09b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:31.776869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f51c91d6-a499-4dd3-b2e5-f7445b1cd09b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:31.778108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f51c91d6-a499-4dd3-b2e5-f7445b1cd09b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:31.843123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:31.887307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 514/999] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:32.179536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b1339-7277-43b8-a50b-8b2e81950af6 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:32.179748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b1339-7277-43b8-a50b-8b2e81950af6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:32.180043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096b1339-7277-43b8-a50b-8b2e81950af6 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:32.181374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 486/1000] 10.222.0.20 () {54 vars in 964 bytes} [Wed Oct 19 21:52:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:32.187240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c4234-ced0-41ec-8dc8-f5d059c3caf5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:32.187524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c4234-ced0-41ec-8dc8-f5d059c3caf5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:32.188356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-630c4234-ced0-41ec-8dc8-f5d059c3caf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:32.262523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-630c4234-ced0-41ec-8dc8-f5d059c3caf5 None None] MFA Rules not processed for user `2f705dc9d2e541f1821dda4b5c7b7709`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:32.263753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-630c4234-ced0-41ec-8dc8-f5d059c3caf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:32.339532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 515/1001] 10.222.0.20 () {58 vars in 1061 bytes} [Wed Oct 19 21:52:32 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:32.348617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096b1339-7277-43b8-a50b-8b2e81950af6 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:32.350241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-096b1339-7277-43b8-a50b-8b2e81950af6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:32.370695 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:32.456614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096b1339-7277-43b8-a50b-8b2e81950af6 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:32.531947 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae6baba-3e0d-406c-ace9-51af9602676c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:32.532551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae6baba-3e0d-406c-ace9-51af9602676c service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:32.532827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae6baba-3e0d-406c-ace9-51af9602676c service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:32.533098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae6baba-3e0d-406c-ace9-51af9602676c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:32.534689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dae6baba-3e0d-406c-ace9-51af9602676c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:32.607579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:32.646695 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae6baba-3e0d-406c-ace9-51af9602676c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:32.648161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae6baba-3e0d-406c-ace9-51af9602676c service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:32.665898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 487/1002] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:35.096378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0813873e-efdc-40bd-9e17-64a300841e63 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:35.096882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0813873e-efdc-40bd-9e17-64a300841e63 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:35.096882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0813873e-efdc-40bd-9e17-64a300841e63 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:35.098191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 516/1003] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:35.103495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb6b9c-fe11-49af-93e1-534c5695c661 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:35.103878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb6b9c-fe11-49af-93e1-534c5695c661 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:35.104053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afb6b9c-fe11-49af-93e1-534c5695c661 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:35.168954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1afb6b9c-fe11-49af-93e1-534c5695c661 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:35.170061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1afb6b9c-fe11-49af-93e1-534c5695c661 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:35.268573 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 488/1004] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:35.547624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0813873e-efdc-40bd-9e17-64a300841e63 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:35.549158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0813873e-efdc-40bd-9e17-64a300841e63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:35.610328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:35.653087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0813873e-efdc-40bd-9e17-64a300841e63 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:35.725342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac4e035-327a-4cc0-a38d-205f3fb76f2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:35.725944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac4e035-327a-4cc0-a38d-205f3fb76f2c service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:35.726220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac4e035-327a-4cc0-a38d-205f3fb76f2c service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:35.726491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac4e035-327a-4cc0-a38d-205f3fb76f2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:35.728375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8ac4e035-327a-4cc0-a38d-205f3fb76f2c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:35.840140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:35.842604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac4e035-327a-4cc0-a38d-205f3fb76f2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:35.844073 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac4e035-327a-4cc0-a38d-205f3fb76f2c service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:35.861709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 517/1005] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:38.100179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba33ad-1578-40e1-b737-70f81e343246 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:38.100704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba33ad-1578-40e1-b737-70f81e343246 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:38.100704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba33ad-1578-40e1-b737-70f81e343246 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:38.102060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 489/1006] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:38.107579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6e892-78c6-41e6-857b-1beb1667c2b9 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:38.108042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6e892-78c6-41e6-857b-1beb1667c2b9 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:38.108146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6e892-78c6-41e6-857b-1beb1667c2b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:38.177520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c3b6e892-78c6-41e6-857b-1beb1667c2b9 None None] MFA Rules not processed for user `7a93405c40b34cf8989eb8d1076eb475`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:38.178753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c3b6e892-78c6-41e6-857b-1beb1667c2b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:38.277455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 518/1007] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:38 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:38.556505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba33ad-1578-40e1-b737-70f81e343246 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:38.558114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7aba33ad-1578-40e1-b737-70f81e343246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:38.590160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:38.673009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba33ad-1578-40e1-b737-70f81e343246 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:38.750300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8601bcf6-2197-4647-801d-913dd30c58a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:38.750906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8601bcf6-2197-4647-801d-913dd30c58a5 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:38.751231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8601bcf6-2197-4647-801d-913dd30c58a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:38.751529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8601bcf6-2197-4647-801d-913dd30c58a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:38.753141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8601bcf6-2197-4647-801d-913dd30c58a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:38.833711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:38.866424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8601bcf6-2197-4647-801d-913dd30c58a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:38.867905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8601bcf6-2197-4647-801d-913dd30c58a5 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:38.887168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 490/1008] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:38 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:41.765258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0a3bc-2a05-473f-9d1b-2c53707ed0c1 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:41.765817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0a3bc-2a05-473f-9d1b-2c53707ed0c1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:41.765817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c0a3bc-2a05-473f-9d1b-2c53707ed0c1 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:41.767214 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 519/1009] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:41.772464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87f16d0-6e39-4768-8895-c275858b4bb0 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:41.772889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87f16d0-6e39-4768-8895-c275858b4bb0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:41.773007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87f16d0-6e39-4768-8895-c275858b4bb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:41.838073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f87f16d0-6e39-4768-8895-c275858b4bb0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:41.839304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f87f16d0-6e39-4768-8895-c275858b4bb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:41.934012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 491/1010] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:52:42.219804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c0a3bc-2a05-473f-9d1b-2c53707ed0c1 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:42.221341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f7c0a3bc-2a05-473f-9d1b-2c53707ed0c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:42.289301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:42.321775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c0a3bc-2a05-473f-9d1b-2c53707ed0c1 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:42.394838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e237ff43-e361-44e3-9dbb-6d68a2dbbdba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:42.395472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237ff43-e361-44e3-9dbb-6d68a2dbbdba service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:42.395747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237ff43-e361-44e3-9dbb-6d68a2dbbdba service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:42.396015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e237ff43-e361-44e3-9dbb-6d68a2dbbdba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:42.397597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e237ff43-e361-44e3-9dbb-6d68a2dbbdba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:42.505802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e237ff43-e361-44e3-9dbb-6d68a2dbbdba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:42.507175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e237ff43-e361-44e3-9dbb-6d68a2dbbdba service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:42.517470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:42.525648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 520/1011] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:44.700595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dbe6ca-ee05-4433-92e7-01233ec66c95 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:44.701144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dbe6ca-ee05-4433-92e7-01233ec66c95 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:44.701144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dbe6ca-ee05-4433-92e7-01233ec66c95 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:44.702490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 492/1012] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:44.707716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cf7727-2b96-46c9-ab35-a4629bc04a7c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:44.708149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cf7727-2b96-46c9-ab35-a4629bc04a7c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:44.708249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8cf7727-2b96-46c9-ab35-a4629bc04a7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:44.772490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d8cf7727-2b96-46c9-ab35-a4629bc04a7c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:44.773685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d8cf7727-2b96-46c9-ab35-a4629bc04a7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:44.867984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 521/1013] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:45.153666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dbe6ca-ee05-4433-92e7-01233ec66c95 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:45.155251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-93dbe6ca-ee05-4433-92e7-01233ec66c95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:45.195569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:45.258607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dbe6ca-ee05-4433-92e7-01233ec66c95 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:45.336857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b5b2f5-c47a-4d8e-880a-2e9c23e682ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:45.337462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b5b2f5-c47a-4d8e-880a-2e9c23e682ee service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:45.337744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b5b2f5-c47a-4d8e-880a-2e9c23e682ee service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:45.338018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b5b2f5-c47a-4d8e-880a-2e9c23e682ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:45.339747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a8b5b2f5-c47a-4d8e-880a-2e9c23e682ee service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:45.422415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:45.447095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b5b2f5-c47a-4d8e-880a-2e9c23e682ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:45.453665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b5b2f5-c47a-4d8e-880a-2e9c23e682ee service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:45.466545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 493/1014] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:48.289674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6f14b-2c33-47f0-a7c8-68a7da7036e0 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:48.290240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6f14b-2c33-47f0-a7c8-68a7da7036e0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:48.290240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c6f14b-2c33-47f0-a7c8-68a7da7036e0 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:48.291609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 522/1015] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:48.296819 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10797376-ca4f-44c5-8482-637fb8eb1f6b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:48.297221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10797376-ca4f-44c5-8482-637fb8eb1f6b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:48.297366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10797376-ca4f-44c5-8482-637fb8eb1f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:48.361597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-10797376-ca4f-44c5-8482-637fb8eb1f6b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:48.362787 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-10797376-ca4f-44c5-8482-637fb8eb1f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:48.457442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 494/1016] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:48.742634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6f14b-2c33-47f0-a7c8-68a7da7036e0 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:48.744205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-57c6f14b-2c33-47f0-a7c8-68a7da7036e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:48.820735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:48.848467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c6f14b-2c33-47f0-a7c8-68a7da7036e0 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:48.920143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1730b006-3123-4266-bdc8-bd9fa78868aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:48.920675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1730b006-3123-4266-bdc8-bd9fa78868aa service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:48.920946 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1730b006-3123-4266-bdc8-bd9fa78868aa service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:48.921212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1730b006-3123-4266-bdc8-bd9fa78868aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:48.923298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1730b006-3123-4266-bdc8-bd9fa78868aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:49.029527 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1730b006-3123-4266-bdc8-bd9fa78868aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:49.030983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1730b006-3123-4266-bdc8-bd9fa78868aa service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:49.051222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 523/1017] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:48 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:53.415513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9eb60d-98b0-4acc-9403-8c78aad61519 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:53.416071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9eb60d-98b0-4acc-9403-8c78aad61519 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:53.416071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9eb60d-98b0-4acc-9403-8c78aad61519 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:53.417386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 495/1018] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:53.422722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c024773-f397-4ede-85c7-e77cfa1fb740 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:53.423163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c024773-f397-4ede-85c7-e77cfa1fb740 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:53.423307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c024773-f397-4ede-85c7-e77cfa1fb740 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:53.428948 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:53.491091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3c024773-f397-4ede-85c7-e77cfa1fb740 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:53.492338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3c024773-f397-4ede-85c7-e77cfa1fb740 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:53.588399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 524/1019] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:53.873665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9eb60d-98b0-4acc-9403-8c78aad61519 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:53.875755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8d9eb60d-98b0-4acc-9403-8c78aad61519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:53.924225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:53.978972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9eb60d-98b0-4acc-9403-8c78aad61519 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:54.053219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8959e271-79d6-49b5-bf6f-404a28dfbc41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:54.053827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959e271-79d6-49b5-bf6f-404a28dfbc41 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:54.054107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959e271-79d6-49b5-bf6f-404a28dfbc41 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:54.054380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8959e271-79d6-49b5-bf6f-404a28dfbc41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:54.056024 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8959e271-79d6-49b5-bf6f-404a28dfbc41 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:54.148504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:54.164361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8959e271-79d6-49b5-bf6f-404a28dfbc41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:54.165796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8959e271-79d6-49b5-bf6f-404a28dfbc41 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:54.184403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 496/1020] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:52:57.279120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c5ccf9-c2de-4cbc-b072-8667da5a3687 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:57.279625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c5ccf9-c2de-4cbc-b072-8667da5a3687 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:57.279715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c5ccf9-c2de-4cbc-b072-8667da5a3687 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:57.280985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 525/1021] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:52:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:52:57.287041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45055662-7965-49c2-baa8-e9143eb84ae2 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:57.287680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45055662-7965-49c2-baa8-e9143eb84ae2 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:57.287838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45055662-7965-49c2-baa8-e9143eb84ae2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:57.359242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-45055662-7965-49c2-baa8-e9143eb84ae2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:52:57.360551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-45055662-7965-49c2-baa8-e9143eb84ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:57.457724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 497/1022] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:52:57 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:52:57.743089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c5ccf9-c2de-4cbc-b072-8667da5a3687 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:52:57.744689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a5c5ccf9-c2de-4cbc-b072-8667da5a3687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:57.837313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:52:57.854147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c5ccf9-c2de-4cbc-b072-8667da5a3687 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:52:57.929515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9180716-50db-45e1-b850-51fbcc7958de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:52:57.930113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9180716-50db-45e1-b850-51fbcc7958de service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:52:57.930386 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9180716-50db-45e1-b850-51fbcc7958de service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:52:57.930904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9180716-50db-45e1-b850-51fbcc7958de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:52:57.932284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a9180716-50db-45e1-b850-51fbcc7958de service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:52:58.057061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9180716-50db-45e1-b850-51fbcc7958de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:52:58.057061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9180716-50db-45e1-b850-51fbcc7958de service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:52:58.074592 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 526/1023] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:52:57 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:02.329953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2f337-c363-411d-b2b6-85fa8c2d0531 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:02.330454 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2f337-c363-411d-b2b6-85fa8c2d0531 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:02.330544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa2f337-c363-411d-b2b6-85fa8c2d0531 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:02.331917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 498/1024] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:02.337663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fb467-6f52-497d-8d7f-f716c89ae054 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:02.338147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fb467-6f52-497d-8d7f-f716c89ae054 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:02.338239 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4fb467-6f52-497d-8d7f-f716c89ae054 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:02.353795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:02.410989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-0c4fb467-6f52-497d-8d7f-f716c89ae054 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:02.412212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0c4fb467-6f52-497d-8d7f-f716c89ae054 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:02.511100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 527/1025] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:02.794634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa2f337-c363-411d-b2b6-85fa8c2d0531 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:02.796214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-efa2f337-c363-411d-b2b6-85fa8c2d0531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:02.854582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:02.900315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa2f337-c363-411d-b2b6-85fa8c2d0531 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:02.976971 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b75e5c1d-489d-4641-b234-faac94b11119 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:02.977572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e5c1d-489d-4641-b234-faac94b11119 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:02.977846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e5c1d-489d-4641-b234-faac94b11119 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:02.978114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b75e5c1d-489d-4641-b234-faac94b11119 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:02.979840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b75e5c1d-489d-4641-b234-faac94b11119 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:03.083069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:03.086571 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75e5c1d-489d-4641-b234-faac94b11119 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:03.088031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b75e5c1d-489d-4641-b234-faac94b11119 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:03.106398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 499/1026] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:08.122529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59845b67-21e0-4cdc-abc3-efbd7101922a None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:08.122983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59845b67-21e0-4cdc-abc3-efbd7101922a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:08.123109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59845b67-21e0-4cdc-abc3-efbd7101922a None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:08.128241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 528/1027] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 21:53:08 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:08.133874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a767be-78e7-4a0b-8130-8e92a786fbf1 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:08.134602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a767be-78e7-4a0b-8130-8e92a786fbf1 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:08.135149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a767be-78e7-4a0b-8130-8e92a786fbf1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:08.232743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-85a767be-78e7-4a0b-8130-8e92a786fbf1 None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:08.234069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-85a767be-78e7-4a0b-8130-8e92a786fbf1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:08.388029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 500/1028] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 21:53:08 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:12.459591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeca472-1a87-4c56-8d34-c021da2256d4 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:12.466413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeca472-1a87-4c56-8d34-c021da2256d4 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:12.466413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eeca472-1a87-4c56-8d34-c021da2256d4 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:12.466413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 529/1029] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 21:53:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:12.467290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a0da8a-a4fd-4e42-a764-6a68f3d082b8 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:12.467750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a0da8a-a4fd-4e42-a764-6a68f3d082b8 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:12.467847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91a0da8a-a4fd-4e42-a764-6a68f3d082b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:12.509445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=59 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:12.525467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:12.548569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-91a0da8a-a4fd-4e42-a764-6a68f3d082b8 None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:12.549785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-91a0da8a-a4fd-4e42-a764-6a68f3d082b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:12.665136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 501/1030] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 21:53:12 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:14.214709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc413bf-390c-4060-b86a-4c5cb4374e32 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:14.215301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc413bf-390c-4060-b86a-4c5cb4374e32 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:14.215410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc413bf-390c-4060-b86a-4c5cb4374e32 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:14.216709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 530/1031] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 21:53:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:14.222408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e05123-643c-4dfe-a6fd-c1e5f273f3bb None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:14.222852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e05123-643c-4dfe-a6fd-c1e5f273f3bb None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:14.223124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e05123-643c-4dfe-a6fd-c1e5f273f3bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:14.299612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-04e05123-643c-4dfe-a6fd-c1e5f273f3bb None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:14.300806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-04e05123-643c-4dfe-a6fd-c1e5f273f3bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:14.341412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:14.407448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 502/1032] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 21:53:14 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:17.884470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd80b5-d1d1-4952-908d-e63e21248666 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:17.885374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd80b5-d1d1-4952-908d-e63e21248666 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:17.885900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bd80b5-d1d1-4952-908d-e63e21248666 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:17.887555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 531/1033] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:17.893926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb799923-01fd-463c-97bb-0b9fefd5bc62 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:17.894415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb799923-01fd-463c-97bb-0b9fefd5bc62 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:17.894498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb799923-01fd-463c-97bb-0b9fefd5bc62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:17.975192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-fb799923-01fd-463c-97bb-0b9fefd5bc62 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:17.976389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fb799923-01fd-463c-97bb-0b9fefd5bc62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:18.079091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 503/1034] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:18.382005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679a51e-5a48-4f7e-a2db-902b62662838 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:18.386110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679a51e-5a48-4f7e-a2db-902b62662838 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:18.386110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679a51e-5a48-4f7e-a2db-902b62662838 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:18.386110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 532/1035] 10.222.0.20 () {54 vars in 961 bytes} [Wed Oct 19 21:53:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:18.394850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c22839-a83d-4504-85ca-fdd94a704be3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:18.395488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c22839-a83d-4504-85ca-fdd94a704be3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:18.395977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c22839-a83d-4504-85ca-fdd94a704be3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:18.418884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:18.485005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a6c22839-a83d-4504-85ca-fdd94a704be3 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:18.486316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a6c22839-a83d-4504-85ca-fdd94a704be3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:18.606698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 504/1036] 10.222.0.20 () {58 vars in 1058 bytes} [Wed Oct 19 21:53:18 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:53:18.615070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679a51e-5a48-4f7e-a2db-902b62662838 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:18.616687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c679a51e-5a48-4f7e-a2db-902b62662838 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:18.790884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679a51e-5a48-4f7e-a2db-902b62662838 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:18.870267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:18.913313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d401a139-e30d-40ca-aa8d-cf522573b9ad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:18.913944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d401a139-e30d-40ca-aa8d-cf522573b9ad service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:18.914186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d401a139-e30d-40ca-aa8d-cf522573b9ad service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:18.914456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d401a139-e30d-40ca-aa8d-cf522573b9ad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:18.916107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d401a139-e30d-40ca-aa8d-cf522573b9ad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:19.010698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d401a139-e30d-40ca-aa8d-cf522573b9ad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:19.012387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d401a139-e30d-40ca-aa8d-cf522573b9ad service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:19.035772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 533/1037] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:21.383243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f544007-ebb0-414e-8ee5-4b1fc5da06d1 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:21.383753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f544007-ebb0-414e-8ee5-4b1fc5da06d1 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:21.383753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f544007-ebb0-414e-8ee5-4b1fc5da06d1 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:21.385210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 505/1038] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:21.390439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9372a152-52bc-421e-b292-cbab8e304b70 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:21.391078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9372a152-52bc-421e-b292-cbab8e304b70 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:21.391078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9372a152-52bc-421e-b292-cbab8e304b70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:21.462116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-9372a152-52bc-421e-b292-cbab8e304b70 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:21.463351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9372a152-52bc-421e-b292-cbab8e304b70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:21.502024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:21.591876 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 534/1039] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:21.888574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f544007-ebb0-414e-8ee5-4b1fc5da06d1 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:21.891022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3f544007-ebb0-414e-8ee5-4b1fc5da06d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:21.963752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:22.051368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f544007-ebb0-414e-8ee5-4b1fc5da06d1 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:22.187706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f684ecdd-9410-4cca-9d34-44f42e060a79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:22.188317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f684ecdd-9410-4cca-9d34-44f42e060a79 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:22.188598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f684ecdd-9410-4cca-9d34-44f42e060a79 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:22.188874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f684ecdd-9410-4cca-9d34-44f42e060a79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:22.190505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f684ecdd-9410-4cca-9d34-44f42e060a79 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:22.215869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:22.292760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f684ecdd-9410-4cca-9d34-44f42e060a79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:22.294569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f684ecdd-9410-4cca-9d34-44f42e060a79 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:22.313119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 506/1040] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 21:53:24.936616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e26ac4-1db0-4472-9ffa-421c43912a49 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:24.937187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e26ac4-1db0-4472-9ffa-421c43912a49 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:24.937386 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e26ac4-1db0-4472-9ffa-421c43912a49 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:24.939256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 535/1041] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:24.944359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b49df5-f023-46eb-8ead-d864d44270f0 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:24.944745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b49df5-f023-46eb-8ead-d864d44270f0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:24.944948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b49df5-f023-46eb-8ead-d864d44270f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:25.014026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f3b49df5-f023-46eb-8ead-d864d44270f0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:25.016024 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f3b49df5-f023-46eb-8ead-d864d44270f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:25.117677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:25.122984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 507/1042] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:25.416941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffddcc82-1307-4662-bb67-33938cafe8d8 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:25.417362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffddcc82-1307-4662-bb67-33938cafe8d8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:25.417743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffddcc82-1307-4662-bb67-33938cafe8d8 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:25.419545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 536/1043] 10.222.0.20 () {54 vars in 961 bytes} [Wed Oct 19 21:53:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:25.425445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0ecd0c-ca63-4e30-a3a0-2e46bb188350 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:25.425839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0ecd0c-ca63-4e30-a3a0-2e46bb188350 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:25.426219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0ecd0c-ca63-4e30-a3a0-2e46bb188350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:25.506290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ea0ecd0c-ca63-4e30-a3a0-2e46bb188350 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:25.507690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ea0ecd0c-ca63-4e30-a3a0-2e46bb188350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:25.631531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 508/1044] 10.222.0.20 () {58 vars in 1058 bytes} [Wed Oct 19 21:53:25 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:25.639751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffddcc82-1307-4662-bb67-33938cafe8d8 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:25.642565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ffddcc82-1307-4662-bb67-33938cafe8d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:25.803326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffddcc82-1307-4662-bb67-33938cafe8d8 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:25.825875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:25.917618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5fa9561-5482-476b-896d-c21dacfc29e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:25.918220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa9561-5482-476b-896d-c21dacfc29e8 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:25.918494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa9561-5482-476b-896d-c21dacfc29e8 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:25.918761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5fa9561-5482-476b-896d-c21dacfc29e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:25.920358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a5fa9561-5482-476b-896d-c21dacfc29e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:26.020279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fa9561-5482-476b-896d-c21dacfc29e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:26.021968 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5fa9561-5482-476b-896d-c21dacfc29e8 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:26.044054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 537/1045] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:26.128796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7ee42-45f6-48c8-8e74-d7174e3e3e30 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:26.129327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7ee42-45f6-48c8-8e74-d7174e3e3e30 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:26.129327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc7ee42-45f6-48c8-8e74-d7174e3e3e30 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:26.130880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 509/1046] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:26.136315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc7b139-b9fe-459e-a5eb-ed0e046da0db None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:26.136604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc7b139-b9fe-459e-a5eb-ed0e046da0db None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:26.136922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc7b139-b9fe-459e-a5eb-ed0e046da0db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:26.169802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:26.209327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-8bc7b139-b9fe-459e-a5eb-ed0e046da0db None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:26.210528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8bc7b139-b9fe-459e-a5eb-ed0e046da0db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:26.320751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 538/1047] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:53:26.658263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ecacd-c2b8-490f-b7e9-77ed006c8796 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:26.658263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ecacd-c2b8-490f-b7e9-77ed006c8796 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:26.658263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8ecacd-c2b8-490f-b7e9-77ed006c8796 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:26.658263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 510/1048] 10.222.0.20 () {54 vars in 963 bytes} [Wed Oct 19 21:53:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:26.662055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fb882c-4ebe-4dcb-9573-4c38d4379f81 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:26.662342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fb882c-4ebe-4dcb-9573-4c38d4379f81 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:26.662616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77fb882c-4ebe-4dcb-9573-4c38d4379f81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:26.744124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-77fb882c-4ebe-4dcb-9573-4c38d4379f81 None None] MFA Rules not processed for user `c065675e9a2f44f59168b9d574e289d5`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:26.745370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-77fb882c-4ebe-4dcb-9573-4c38d4379f81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:26.800321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:26.832320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 539/1049] 10.222.0.20 () {58 vars in 1060 bytes} [Wed Oct 19 21:53:26 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:53:26.840036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ecacd-c2b8-490f-b7e9-77ed006c8796 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:26.841683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6e8ecacd-c2b8-490f-b7e9-77ed006c8796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:26.893694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:26.949188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8ecacd-c2b8-490f-b7e9-77ed006c8796 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:27.032267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e468bf-b22c-4ee2-bced-d94be8f35ebb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:27.033244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e468bf-b22c-4ee2-bced-d94be8f35ebb service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:27.033641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e468bf-b22c-4ee2-bced-d94be8f35ebb service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:27.033981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e468bf-b22c-4ee2-bced-d94be8f35ebb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:27.035879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-85e468bf-b22c-4ee2-bced-d94be8f35ebb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:27.136348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:27.149861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e468bf-b22c-4ee2-bced-d94be8f35ebb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:27.153219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e468bf-b22c-4ee2-bced-d94be8f35ebb service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:27.173921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 511/1050] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 21:53:28.433496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae02e09-3d00-436d-a3ed-6347abd04652 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:28.434041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae02e09-3d00-436d-a3ed-6347abd04652 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:28.434041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae02e09-3d00-436d-a3ed-6347abd04652 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:28.435396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 540/1051] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:28.440741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59c36a-d13c-4948-9957-e0d3e27d7a28 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:28.441160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59c36a-d13c-4948-9957-e0d3e27d7a28 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:28.441326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e59c36a-d13c-4948-9957-e0d3e27d7a28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:28.509062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6e59c36a-d13c-4948-9957-e0d3e27d7a28 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:28.510257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6e59c36a-d13c-4948-9957-e0d3e27d7a28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:28.608231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 512/1052] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:28.895122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae02e09-3d00-436d-a3ed-6347abd04652 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:28.896752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2ae02e09-3d00-436d-a3ed-6347abd04652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:29.049482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae02e09-3d00-436d-a3ed-6347abd04652 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:29.105420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:29.171824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb9c0a8-249c-48bd-b4ad-f3b17d6d0e19 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:29.172431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb9c0a8-249c-48bd-b4ad-f3b17d6d0e19 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:29.172708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb9c0a8-249c-48bd-b4ad-f3b17d6d0e19 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:29.172978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb9c0a8-249c-48bd-b4ad-f3b17d6d0e19 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:29.174574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cbb9c0a8-249c-48bd-b4ad-f3b17d6d0e19 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:29.270790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb9c0a8-249c-48bd-b4ad-f3b17d6d0e19 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:29.272315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb9c0a8-249c-48bd-b4ad-f3b17d6d0e19 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:29.291375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 541/1053] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:31.626404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96d34cd-f664-49b1-971f-6c5ef883e673 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:31.626985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96d34cd-f664-49b1-971f-6c5ef883e673 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:31.626985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96d34cd-f664-49b1-971f-6c5ef883e673 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:31.628457 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 513/1054] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:31.633574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92790794-d0dd-4a7b-85c3-bb22cc4e73d5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:31.633987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92790794-d0dd-4a7b-85c3-bb22cc4e73d5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:31.634153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92790794-d0dd-4a7b-85c3-bb22cc4e73d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:31.702124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-92790794-d0dd-4a7b-85c3-bb22cc4e73d5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:31.703885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-92790794-d0dd-4a7b-85c3-bb22cc4e73d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:31.711351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:31.808678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 542/1055] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:32.095570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96d34cd-f664-49b1-971f-6c5ef883e673 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:32.097119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f96d34cd-f664-49b1-971f-6c5ef883e673 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:32.162138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:32.252948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96d34cd-f664-49b1-971f-6c5ef883e673 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:32.375176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f6a7f-2c23-47e9-be90-78f7d508f705 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:32.375699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f6a7f-2c23-47e9-be90-78f7d508f705 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:32.376103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f6a7f-2c23-47e9-be90-78f7d508f705 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:32.376373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f6a7f-2c23-47e9-be90-78f7d508f705 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:32.378065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ec4f6a7f-2c23-47e9-be90-78f7d508f705 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:32.398078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:32.475833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f6a7f-2c23-47e9-be90-78f7d508f705 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:32.477424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f6a7f-2c23-47e9-be90-78f7d508f705 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:32.497204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 514/1056] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:34.859014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47393d65-a73c-4f6a-96a1-2c0a2e41ea03 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:34.859435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47393d65-a73c-4f6a-96a1-2c0a2e41ea03 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:34.859656 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47393d65-a73c-4f6a-96a1-2c0a2e41ea03 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:34.862830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 543/1057] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:34.866500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab20040-d8eb-44d8-adc7-c0982a5970ed None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:34.866981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab20040-d8eb-44d8-adc7-c0982a5970ed None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:34.867242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab20040-d8eb-44d8-adc7-c0982a5970ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:34.946446 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-eab20040-d8eb-44d8-adc7-c0982a5970ed None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:34.947696 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-eab20040-d8eb-44d8-adc7-c0982a5970ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:35.041001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:35.055142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 515/1058] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:53:35.344702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47393d65-a73c-4f6a-96a1-2c0a2e41ea03 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:35.346217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-47393d65-a73c-4f6a-96a1-2c0a2e41ea03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:35.487368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:35.500307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47393d65-a73c-4f6a-96a1-2c0a2e41ea03 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:35.622099 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa4d8f4-ee37-432e-9b65-4a9036457308 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:35.622832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa4d8f4-ee37-432e-9b65-4a9036457308 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:35.623203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa4d8f4-ee37-432e-9b65-4a9036457308 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:35.623545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa4d8f4-ee37-432e-9b65-4a9036457308 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:35.626280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-daa4d8f4-ee37-432e-9b65-4a9036457308 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:35.730832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa4d8f4-ee37-432e-9b65-4a9036457308 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:35.732469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daa4d8f4-ee37-432e-9b65-4a9036457308 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:35.743514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:35.752112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 544/1059] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:38.102162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d06dc-6fdd-4014-beb8-bd33be1d1150 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:38.102674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d06dc-6fdd-4014-beb8-bd33be1d1150 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:38.102674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3d06dc-6fdd-4014-beb8-bd33be1d1150 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:38.104032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 516/1060] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:38.109458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a8921-83f4-428a-8e4a-ea22806a8e1c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:38.109936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a8921-83f4-428a-8e4a-ea22806a8e1c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:38.110095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8a8921-83f4-428a-8e4a-ea22806a8e1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:38.177648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6d8a8921-83f4-428a-8e4a-ea22806a8e1c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:38.178871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6d8a8921-83f4-428a-8e4a-ea22806a8e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:38.282789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 545/1061] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:53:38.579926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3d06dc-6fdd-4014-beb8-bd33be1d1150 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:38.581509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6c3d06dc-6fdd-4014-beb8-bd33be1d1150 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:38.736853 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3d06dc-6fdd-4014-beb8-bd33be1d1150 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:38.814241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:38.865339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f458558d-ec45-445c-8a9f-a59f75bb82b0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:38.865881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458558d-ec45-445c-8a9f-a59f75bb82b0 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:38.866168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458558d-ec45-445c-8a9f-a59f75bb82b0 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:38.866457 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f458558d-ec45-445c-8a9f-a59f75bb82b0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:38.868207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f458558d-ec45-445c-8a9f-a59f75bb82b0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:38.981363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f458558d-ec45-445c-8a9f-a59f75bb82b0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:38.983133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f458558d-ec45-445c-8a9f-a59f75bb82b0 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:39.004354 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 517/1062] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:41.211068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10de5fdc-acb5-42d7-80b8-eb448a604afa None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:41.211680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10de5fdc-acb5-42d7-80b8-eb448a604afa None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:41.211680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10de5fdc-acb5-42d7-80b8-eb448a604afa None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:41.212934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 546/1063] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:41.218385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd90f15-949b-4b10-8af5-17393ccb48d3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:41.218797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd90f15-949b-4b10-8af5-17393ccb48d3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:41.219041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd90f15-949b-4b10-8af5-17393ccb48d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:41.284491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8cd90f15-949b-4b10-8af5-17393ccb48d3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:41.285702 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8cd90f15-949b-4b10-8af5-17393ccb48d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:41.318534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:41.385480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 518/1064] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:41.673096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10de5fdc-acb5-42d7-80b8-eb448a604afa None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:41.674594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-10de5fdc-acb5-42d7-80b8-eb448a604afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:41.751093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:41.824521 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10de5fdc-acb5-42d7-80b8-eb448a604afa None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:41.938618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b794d6f-0468-4c5c-9e20-da2664617f52 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:41.939212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b794d6f-0468-4c5c-9e20-da2664617f52 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:41.939516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b794d6f-0468-4c5c-9e20-da2664617f52 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:41.939816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b794d6f-0468-4c5c-9e20-da2664617f52 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:41.941446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3b794d6f-0468-4c5c-9e20-da2664617f52 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:41.980484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:42.032683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b794d6f-0468-4c5c-9e20-da2664617f52 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:42.034367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b794d6f-0468-4c5c-9e20-da2664617f52 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:42.052752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 547/1065] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:42.614045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5f9ef-2103-4c2e-b4f7-63c2115ea7ae None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:42.614576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5f9ef-2103-4c2e-b4f7-63c2115ea7ae None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:42.614576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5f9ef-2103-4c2e-b4f7-63c2115ea7ae None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:42.615875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 519/1066] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:42.621071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63588bb2-6518-447d-b892-c0ef6b918901 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:42.621484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63588bb2-6518-447d-b892-c0ef6b918901 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:42.621710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63588bb2-6518-447d-b892-c0ef6b918901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:42.691128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-63588bb2-6518-447d-b892-c0ef6b918901 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:42.692320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-63588bb2-6518-447d-b892-c0ef6b918901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:42.792509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 548/1067] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:43.089700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569be530-e810-45b9-8ffd-ae3e5338cb0f None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:43.090019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569be530-e810-45b9-8ffd-ae3e5338cb0f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:43.090324 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569be530-e810-45b9-8ffd-ae3e5338cb0f None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:43.091729 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 520/1068] 10.222.0.20 () {54 vars in 962 bytes} [Wed Oct 19 21:53:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:43.097107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a0bf76-c00c-441f-b3bd-f8088ee9b032 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:43.097474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a0bf76-c00c-441f-b3bd-f8088ee9b032 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:43.097904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a0bf76-c00c-441f-b3bd-f8088ee9b032 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:43.114288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:43.174587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-09a0bf76-c00c-441f-b3bd-f8088ee9b032 None None] MFA Rules not processed for user `9ebc359d9bf54edcb9634afc31cf188a`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:43.175811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-09a0bf76-c00c-441f-b3bd-f8088ee9b032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:43.250374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 549/1069] 10.222.0.20 () {58 vars in 1059 bytes} [Wed Oct 19 21:53:43 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:43.259570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569be530-e810-45b9-8ffd-ae3e5338cb0f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:43.261083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-569be530-e810-45b9-8ffd-ae3e5338cb0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:43.364436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569be530-e810-45b9-8ffd-ae3e5338cb0f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:43.421872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:43.438499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d24be-1402-4e2d-80dd-9bd83108c04d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:43.439125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d24be-1402-4e2d-80dd-9bd83108c04d service glance] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:43.439402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d24be-1402-4e2d-80dd-9bd83108c04d service glance] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:43.439691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d24be-1402-4e2d-80dd-9bd83108c04d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:43.441263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b48d24be-1402-4e2d-80dd-9bd83108c04d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:43.549734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48d24be-1402-4e2d-80dd-9bd83108c04d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:43.551202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48d24be-1402-4e2d-80dd-9bd83108c04d service glance] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:43.569082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 521/1070] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:44.352067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bda6e0-60fa-4b9b-8f6f-9987f0e06e9f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:44.359066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bda6e0-60fa-4b9b-8f6f-9987f0e06e9f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:44.359066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bda6e0-60fa-4b9b-8f6f-9987f0e06e9f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:44.359066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 550/1071] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:44.359359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fe6d5c-55df-430a-8293-c1170e42f145 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:44.360250 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fe6d5c-55df-430a-8293-c1170e42f145 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:44.360420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31fe6d5c-55df-430a-8293-c1170e42f145 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:44.426921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-31fe6d5c-55df-430a-8293-c1170e42f145 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:44.428153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-31fe6d5c-55df-430a-8293-c1170e42f145 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:44.450798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ecade0-1845-4b26-8726-11d665833ffc None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:44.456542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ecade0-1845-4b26-8726-11d665833ffc None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:44.456542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ecade0-1845-4b26-8726-11d665833ffc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:44.474127 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:44.523462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-06ecade0-1845-4b26-8726-11d665833ffc None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:44.524681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-06ecade0-1845-4b26-8726-11d665833ffc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:44.531012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 522/1072] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:44.581781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:44.612970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 551/1073] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 21:53:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:53:44.656176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89597ab5-0d0e-407e-b6f0-c1ad9fa17520 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:44.656493 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89597ab5-0d0e-407e-b6f0-c1ad9fa17520 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:44.656840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89597ab5-0d0e-407e-b6f0-c1ad9fa17520 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:44.658225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 523/1074] 10.222.0.20 () {54 vars in 943 bytes} [Wed Oct 19 21:53:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:44.666328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c624546f-1696-4a3d-b3bc-7f94d693661a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:44.666632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c624546f-1696-4a3d-b3bc-7f94d693661a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:44.666903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c624546f-1696-4a3d-b3bc-7f94d693661a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:44.744613 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c624546f-1696-4a3d-b3bc-7f94d693661a None None] MFA Rules not processed for user `b19bbdff6ae1428d8cfc7a33d4fafb49`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:44.745792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c624546f-1696-4a3d-b3bc-7f94d693661a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:44.826863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89597ab5-0d0e-407e-b6f0-c1ad9fa17520 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:44.828402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-89597ab5-0d0e-407e-b6f0-c1ad9fa17520 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:44.857590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 552/1075] 10.222.0.20 () {58 vars in 1040 bytes} [Wed Oct 19 21:53:44 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:44.867400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c624546f-1696-4a3d-b3bc-7f94d693661a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:44.876836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c624546f-1696-4a3d-b3bc-7f94d693661a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:44.902678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:44.977551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89597ab5-0d0e-407e-b6f0-c1ad9fa17520 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:44.981231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00c650c-643a-4b81-937c-a7b5ebc02c37 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:44.981831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00c650c-643a-4b81-937c-a7b5ebc02c37 service swift] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:44.982103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00c650c-643a-4b81-937c-a7b5ebc02c37 service swift] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:44.982370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00c650c-643a-4b81-937c-a7b5ebc02c37 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:44.984024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d00c650c-643a-4b81-937c-a7b5ebc02c37 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:45.005268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:45.069903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00c650c-643a-4b81-937c-a7b5ebc02c37 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:45.071534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00c650c-643a-4b81-937c-a7b5ebc02c37 service swift] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:45.080078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 553/1076] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 21:53:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 19 21:53:45.095906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b24c5e-1579-4794-9e71-6c9f7b2baadf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:45.096523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b24c5e-1579-4794-9e71-6c9f7b2baadf service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:45.096802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b24c5e-1579-4794-9e71-6c9f7b2baadf service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:45.097076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b24c5e-1579-4794-9e71-6c9f7b2baadf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:45.098692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a8b24c5e-1579-4794-9e71-6c9f7b2baadf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:45.189309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b24c5e-1579-4794-9e71-6c9f7b2baadf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:45.190845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b24c5e-1579-4794-9e71-6c9f7b2baadf service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:45.208693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 524/1077] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:47.574497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f899ca8-a4d8-41c5-bedd-f24dc9f73dc8 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:47.575069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f899ca8-a4d8-41c5-bedd-f24dc9f73dc8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:47.575069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f899ca8-a4d8-41c5-bedd-f24dc9f73dc8 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:47.580607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 554/1078] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:47 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:47.582187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df2873-22d0-403d-8d26-64bd90f76283 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:47.582635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df2873-22d0-403d-8d26-64bd90f76283 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:47.582732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2df2873-22d0-403d-8d26-64bd90f76283 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:47.624558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:47.660584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f2df2873-22d0-403d-8d26-64bd90f76283 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:47.661931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f2df2873-22d0-403d-8d26-64bd90f76283 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:47.764148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 525/1079] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:53:48.058693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f899ca8-a4d8-41c5-bedd-f24dc9f73dc8 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:48.060280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6f899ca8-a4d8-41c5-bedd-f24dc9f73dc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:48.132358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:48.226763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f899ca8-a4d8-41c5-bedd-f24dc9f73dc8 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:48.371466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784e386e-5474-442e-b65e-be4be5ab0b04 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:48.372228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e386e-5474-442e-b65e-be4be5ab0b04 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:48.372507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e386e-5474-442e-b65e-be4be5ab0b04 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:48.372825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e386e-5474-442e-b65e-be4be5ab0b04 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:48.374915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-784e386e-5474-442e-b65e-be4be5ab0b04 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:48.398377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:48.480789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784e386e-5474-442e-b65e-be4be5ab0b04 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:48.482795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784e386e-5474-442e-b65e-be4be5ab0b04 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:48.503884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 555/1080] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 445 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:50.846508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae651399-9379-41a1-8704-21ff10dae57a None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:50.847077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae651399-9379-41a1-8704-21ff10dae57a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:50.847183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae651399-9379-41a1-8704-21ff10dae57a None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:50.848581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 526/1081] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:50.853906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef32c11-a144-454f-b742-1c85e28ab978 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:50.854359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef32c11-a144-454f-b742-1c85e28ab978 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:50.854575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef32c11-a144-454f-b742-1c85e28ab978 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:50.942214 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-fef32c11-a144-454f-b742-1c85e28ab978 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:50.943516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fef32c11-a144-454f-b742-1c85e28ab978 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:51.063337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:51.070162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 556/1082] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:51.360544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae651399-9379-41a1-8704-21ff10dae57a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:51.362987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ae651399-9379-41a1-8704-21ff10dae57a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:51.481175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:51.530815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae651399-9379-41a1-8704-21ff10dae57a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:51.663358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75614ab-c35a-4f64-b402-88401b3a38e2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:51.664560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75614ab-c35a-4f64-b402-88401b3a38e2 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:51.664844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75614ab-c35a-4f64-b402-88401b3a38e2 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:51.665113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75614ab-c35a-4f64-b402-88401b3a38e2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:51.666729 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a75614ab-c35a-4f64-b402-88401b3a38e2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:51.748551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:51.768037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75614ab-c35a-4f64-b402-88401b3a38e2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:51.769755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75614ab-c35a-4f64-b402-88401b3a38e2 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:51.796651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 527/1083] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:53:54.095807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413f9c82-60f1-493e-9d97-b8c93de2a146 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:54.096320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413f9c82-60f1-493e-9d97-b8c93de2a146 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:54.096320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413f9c82-60f1-493e-9d97-b8c93de2a146 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:54.101726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 557/1084] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:53:54 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:53:54.103239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f111c-604b-4483-91bd-a25f7d30d97d None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:54.103671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f111c-604b-4483-91bd-a25f7d30d97d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:54.103770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f111c-604b-4483-91bd-a25f7d30d97d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:54.185398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-4e1f111c-604b-4483-91bd-a25f7d30d97d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:53:54.186744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4e1f111c-604b-4483-91bd-a25f7d30d97d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:54.298163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 528/1085] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:53:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:53:54.591874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413f9c82-60f1-493e-9d97-b8c93de2a146 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:53:54.593388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-413f9c82-60f1-493e-9d97-b8c93de2a146 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:54.761634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413f9c82-60f1-493e-9d97-b8c93de2a146 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:53:54.842025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:53:54.891070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97765be3-a520-455d-8b65-834ad452f59a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:53:54.891680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97765be3-a520-455d-8b65-834ad452f59a service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:53:54.891963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97765be3-a520-455d-8b65-834ad452f59a service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:53:54.892240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97765be3-a520-455d-8b65-834ad452f59a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:53:54.893854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-97765be3-a520-455d-8b65-834ad452f59a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:53:54.988220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97765be3-a520-455d-8b65-834ad452f59a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:53:54.989870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97765be3-a520-455d-8b65-834ad452f59a service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:53:55.009589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 558/1086] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:53:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:04.296467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=79 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:05.007152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=64 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:08.333628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45dec5-4c7b-48d1-99e0-15649e455305 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:08.334155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45dec5-4c7b-48d1-99e0-15649e455305 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:08.334261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c45dec5-4c7b-48d1-99e0-15649e455305 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:08.335628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 529/1087] 10.222.0.20 () {54 vars in 918 bytes} [Wed Oct 19 21:54:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:08.340893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7a8d2-22a3-446c-b32a-9dcb3d9e6e48 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:08.341299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7a8d2-22a3-446c-b32a-9dcb3d9e6e48 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:08.341438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad7a8d2-22a3-446c-b32a-9dcb3d9e6e48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:08.418507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-aad7a8d2-22a3-446c-b32a-9dcb3d9e6e48 None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:08.419787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-aad7a8d2-22a3-446c-b32a-9dcb3d9e6e48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:08.524194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 559/1088] 10.222.0.20 () {58 vars in 1015 bytes} [Wed Oct 19 21:54:08 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:08.538420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac3c8fe-888f-4851-bab3-d2f47ad682c9 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:08.538701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac3c8fe-888f-4851-bab3-d2f47ad682c9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:08.539273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac3c8fe-888f-4851-bab3-d2f47ad682c9 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:08.540590 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 530/1089] 10.222.0.20 () {54 vars in 918 bytes} [Wed Oct 19 21:54:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:08.545305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4fbc05-d48b-462e-8e1d-6967b33101f9 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:08.545587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4fbc05-d48b-462e-8e1d-6967b33101f9 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:08.545856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4fbc05-d48b-462e-8e1d-6967b33101f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:08.617885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:08.620827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3b4fbc05-d48b-462e-8e1d-6967b33101f9 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:08.622018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3b4fbc05-d48b-462e-8e1d-6967b33101f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:08.736230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 560/1090] 10.222.0.20 () {58 vars in 1015 bytes} [Wed Oct 19 21:54:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:54:11.293731 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94d3fad-451f-4696-857b-8697f91ed717 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:11.294294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94d3fad-451f-4696-857b-8697f91ed717 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:11.294294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94d3fad-451f-4696-857b-8697f91ed717 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:11.295651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 531/1091] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:11.301363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372c015c-dde7-47c4-bf1c-6a5bdbc3ceae None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:11.301797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372c015c-dde7-47c4-bf1c-6a5bdbc3ceae None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:11.301903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372c015c-dde7-47c4-bf1c-6a5bdbc3ceae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:11.368806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-372c015c-dde7-47c4-bf1c-6a5bdbc3ceae None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:11.370017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-372c015c-dde7-47c4-bf1c-6a5bdbc3ceae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:11.425162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:11.467165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 561/1092] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:11.753986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94d3fad-451f-4696-857b-8697f91ed717 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:11.755545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a94d3fad-451f-4696-857b-8697f91ed717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:11.900490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94d3fad-451f-4696-857b-8697f91ed717 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:11.972400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:12.012770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ba87f7-ea70-4592-b508-57c5e2f1492e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:12.013300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba87f7-ea70-4592-b508-57c5e2f1492e service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:12.013577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba87f7-ea70-4592-b508-57c5e2f1492e service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:12.013850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ba87f7-ea70-4592-b508-57c5e2f1492e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:12.015469 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-92ba87f7-ea70-4592-b508-57c5e2f1492e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:12.102598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ba87f7-ea70-4592-b508-57c5e2f1492e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:12.104217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ba87f7-ea70-4592-b508-57c5e2f1492e service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:12.122111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 532/1093] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:13.252045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6a03c-a31c-4cd8-bae8-a394f92c9e99 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:13.252593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6a03c-a31c-4cd8-bae8-a394f92c9e99 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:13.252593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6a03c-a31c-4cd8-bae8-a394f92c9e99 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:13.253908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 562/1094] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 21:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:13.260103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454d7fff-b5c2-4c9d-b723-711a6bca934f None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:13.260530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454d7fff-b5c2-4c9d-b723-711a6bca934f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:13.260782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454d7fff-b5c2-4c9d-b723-711a6bca934f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:13.340828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-454d7fff-b5c2-4c9d-b723-711a6bca934f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:13.342063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-454d7fff-b5c2-4c9d-b723-711a6bca934f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:13.369428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:13.438023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 533/1095] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 21:54:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:13.587735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e6a03c-a31c-4cd8-bae8-a394f92c9e99 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:13.589190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a6e6a03c-a31c-4cd8-bae8-a394f92c9e99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:13.732874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e6a03c-a31c-4cd8-bae8-a394f92c9e99 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:13.774500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:13.846860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af7ac5c-c297-40e5-a9ed-1a603aab0972 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:13.852434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af7ac5c-c297-40e5-a9ed-1a603aab0972 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:13.852434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af7ac5c-c297-40e5-a9ed-1a603aab0972 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:13.852434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af7ac5c-c297-40e5-a9ed-1a603aab0972 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:13.852434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6af7ac5c-c297-40e5-a9ed-1a603aab0972 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:13.938652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af7ac5c-c297-40e5-a9ed-1a603aab0972 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:13.940322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af7ac5c-c297-40e5-a9ed-1a603aab0972 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:13.958843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 563/1096] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:15.087860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678cb61-7fbd-4d42-888f-b9a41fffa552 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:15.088400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678cb61-7fbd-4d42-888f-b9a41fffa552 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:15.088523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c678cb61-7fbd-4d42-888f-b9a41fffa552 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:15.089772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 534/1097] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 21:54:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:15.096465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f8fde-3307-416e-83cd-1e8e6035bfaf None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:15.096909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f8fde-3307-416e-83cd-1e8e6035bfaf None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:15.097006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982f8fde-3307-416e-83cd-1e8e6035bfaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:15.150368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:15.172048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-982f8fde-3307-416e-83cd-1e8e6035bfaf None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:15.173263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-982f8fde-3307-416e-83cd-1e8e6035bfaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:15.268450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 564/1098] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 21:54:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:15.419868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678cb61-7fbd-4d42-888f-b9a41fffa552 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:15.421359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c678cb61-7fbd-4d42-888f-b9a41fffa552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:15.566418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c678cb61-7fbd-4d42-888f-b9a41fffa552 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:15.578013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:15.679081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b77d50-5db0-4b96-9964-771367727f72 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:15.679611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b77d50-5db0-4b96-9964-771367727f72 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:15.679884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b77d50-5db0-4b96-9964-771367727f72 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:15.680151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b77d50-5db0-4b96-9964-771367727f72 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:15.681729 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-31b77d50-5db0-4b96-9964-771367727f72 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:15.768268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b77d50-5db0-4b96-9964-771367727f72 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:15.769848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b77d50-5db0-4b96-9964-771367727f72 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:15.787514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 535/1099] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:16.899287 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d4b01-0a09-4b40-a707-ae570dfd61dc None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:16.899914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d4b01-0a09-4b40-a707-ae570dfd61dc None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:16.899914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197d4b01-0a09-4b40-a707-ae570dfd61dc None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:16.901168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 565/1100] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 21:54:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:16.907417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e310a5-ccb7-47cf-be48-57c69e01e376 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:16.907839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e310a5-ccb7-47cf-be48-57c69e01e376 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:16.907960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e310a5-ccb7-47cf-be48-57c69e01e376 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:16.928154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:16.983982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e3e310a5-ccb7-47cf-be48-57c69e01e376 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:16.985746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e3e310a5-ccb7-47cf-be48-57c69e01e376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:17.079783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 536/1101] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 21:54:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:17.230157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d4b01-0a09-4b40-a707-ae570dfd61dc None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:17.231736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-197d4b01-0a09-4b40-a707-ae570dfd61dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:17.341336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:17.377196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197d4b01-0a09-4b40-a707-ae570dfd61dc None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:17.487968 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7924085-9936-44f1-9cc2-323a54434e7c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:17.488573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7924085-9936-44f1-9cc2-323a54434e7c service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:17.488851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7924085-9936-44f1-9cc2-323a54434e7c service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:17.489123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7924085-9936-44f1-9cc2-323a54434e7c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:17.490732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e7924085-9936-44f1-9cc2-323a54434e7c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:17.567846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:17.580888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7924085-9936-44f1-9cc2-323a54434e7c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:17.582474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7924085-9936-44f1-9cc2-323a54434e7c service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:17.600227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 566/1102] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:18.702543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b55f46f-ac06-455f-881c-7a94b424e61f None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:18.703045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b55f46f-ac06-455f-881c-7a94b424e61f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:18.703143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b55f46f-ac06-455f-881c-7a94b424e61f None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:18.704494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 537/1103] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 21:54:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:18.710878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2703f41f-62b5-4d38-a567-e935ba2de977 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:18.711326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2703f41f-62b5-4d38-a567-e935ba2de977 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:18.711461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2703f41f-62b5-4d38-a567-e935ba2de977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:18.795357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2703f41f-62b5-4d38-a567-e935ba2de977 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:18.796596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2703f41f-62b5-4d38-a567-e935ba2de977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:18.896097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 567/1104] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 21:54:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:19.050094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b55f46f-ac06-455f-881c-7a94b424e61f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:19.051799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2b55f46f-ac06-455f-881c-7a94b424e61f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:19.134938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:19.196861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b55f46f-ac06-455f-881c-7a94b424e61f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:19.307001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cfac8e-e6f4-4b9c-b85a-e319891ae937 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:19.307509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cfac8e-e6f4-4b9c-b85a-e319891ae937 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:19.307786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cfac8e-e6f4-4b9c-b85a-e319891ae937 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:19.308056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cfac8e-e6f4-4b9c-b85a-e319891ae937 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:19.309651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-44cfac8e-e6f4-4b9c-b85a-e319891ae937 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:19.357152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:19.396727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cfac8e-e6f4-4b9c-b85a-e319891ae937 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:19.398318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cfac8e-e6f4-4b9c-b85a-e319891ae937 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:19.416455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 538/1105] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:20.508671 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0b6f33-ab2c-426d-b1fa-959d076faa67 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:20.509322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0b6f33-ab2c-426d-b1fa-959d076faa67 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:20.509322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0b6f33-ab2c-426d-b1fa-959d076faa67 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:20.510574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 568/1106] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 21:54:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:20.517224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3804c-ec49-455e-aaf1-77587fdf9517 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:20.517655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3804c-ec49-455e-aaf1-77587fdf9517 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:20.517769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a3804c-ec49-455e-aaf1-77587fdf9517 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:20.592689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-70a3804c-ec49-455e-aaf1-77587fdf9517 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:20.594498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-70a3804c-ec49-455e-aaf1-77587fdf9517 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:20.688090 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 539/1107] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 21:54:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:20.837829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0b6f33-ab2c-426d-b1fa-959d076faa67 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:20.839524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5b0b6f33-ab2c-426d-b1fa-959d076faa67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:20.887786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:20.981537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0b6f33-ab2c-426d-b1fa-959d076faa67 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:21.090855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045a020f-6661-4135-8951-c158d770f511 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:21.091401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045a020f-6661-4135-8951-c158d770f511 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:21.091682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045a020f-6661-4135-8951-c158d770f511 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:21.091952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045a020f-6661-4135-8951-c158d770f511 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:21.093516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-045a020f-6661-4135-8951-c158d770f511 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:21.100213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:21.180646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045a020f-6661-4135-8951-c158d770f511 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:21.182243 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045a020f-6661-4135-8951-c158d770f511 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:21.200190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 569/1108] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:22.313111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdb0d2-e3d9-4ad2-840c-09c8a0cfb78a None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:22.313685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdb0d2-e3d9-4ad2-840c-09c8a0cfb78a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:22.313685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bdb0d2-e3d9-4ad2-840c-09c8a0cfb78a None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:22.314995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 540/1109] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 21:54:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:22.321047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f1be8-3d60-430a-ba7f-9399328d5c01 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:22.321481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f1be8-3d60-430a-ba7f-9399328d5c01 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:22.321598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f1be8-3d60-430a-ba7f-9399328d5c01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:22.402712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2a1f1be8-3d60-430a-ba7f-9399328d5c01 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:22.403956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2a1f1be8-3d60-430a-ba7f-9399328d5c01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:22.470759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:22.498814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 570/1110] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 21:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:22.649025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bdb0d2-e3d9-4ad2-840c-09c8a0cfb78a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:22.650493 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-11bdb0d2-e3d9-4ad2-840c-09c8a0cfb78a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:22.668359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:22.795143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bdb0d2-e3d9-4ad2-840c-09c8a0cfb78a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:22.881419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:22.906421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9623d0-f7ab-4e1a-a370-4d88849a5bf5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:22.907042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9623d0-f7ab-4e1a-a370-4d88849a5bf5 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:22.907363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9623d0-f7ab-4e1a-a370-4d88849a5bf5 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:22.907675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9623d0-f7ab-4e1a-a370-4d88849a5bf5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:22.909366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3c9623d0-f7ab-4e1a-a370-4d88849a5bf5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:22.999375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9623d0-f7ab-4e1a-a370-4d88849a5bf5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:23.001034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9623d0-f7ab-4e1a-a370-4d88849a5bf5 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:23.018562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 541/1111] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:24.114241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db364b0-dc1c-483a-af09-a7c1129f3c1a None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:24.114758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db364b0-dc1c-483a-af09-a7c1129f3c1a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:24.114758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db364b0-dc1c-483a-af09-a7c1129f3c1a None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:24.116197 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 571/1112] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 21:54:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:24.122459 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53d75fe-af76-4150-b88b-fb436fbb4590 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:24.125601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53d75fe-af76-4150-b88b-fb436fbb4590 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:24.125601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53d75fe-af76-4150-b88b-fb436fbb4590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:24.199686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a53d75fe-af76-4150-b88b-fb436fbb4590 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:24.200877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a53d75fe-af76-4150-b88b-fb436fbb4590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:24.240891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:24.294871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 542/1113] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 21:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:24.448358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db364b0-dc1c-483a-af09-a7c1129f3c1a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:24.449939 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8db364b0-dc1c-483a-af09-a7c1129f3c1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:24.593868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db364b0-dc1c-483a-af09-a7c1129f3c1a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:24.647506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:24.706536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f9214c-b92b-4243-a9c4-286b2c159b51 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:24.707175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9214c-b92b-4243-a9c4-286b2c159b51 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:24.707458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9214c-b92b-4243-a9c4-286b2c159b51 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:24.707729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f9214c-b92b-4243-a9c4-286b2c159b51 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:24.709315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-67f9214c-b92b-4243-a9c4-286b2c159b51 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:24.797915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f9214c-b92b-4243-a9c4-286b2c159b51 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:24.799543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f9214c-b92b-4243-a9c4-286b2c159b51 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:24.817227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 572/1114] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:25.919078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d521db7-7747-497c-9522-d4e5615cdfa9 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:25.919657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d521db7-7747-497c-9522-d4e5615cdfa9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:25.919657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d521db7-7747-497c-9522-d4e5615cdfa9 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:25.920948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 543/1115] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 21:54:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:25.927786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c9bd79-da69-4ebc-9f1c-5c20c6441506 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:25.928199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c9bd79-da69-4ebc-9f1c-5c20c6441506 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:25.928339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c9bd79-da69-4ebc-9f1c-5c20c6441506 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:26.003607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b1c9bd79-da69-4ebc-9f1c-5c20c6441506 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:26.004812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b1c9bd79-da69-4ebc-9f1c-5c20c6441506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:26.006794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:26.103937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 573/1116] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 21:54:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:26.252780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d521db7-7747-497c-9522-d4e5615cdfa9 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:26.254316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5d521db7-7747-497c-9522-d4e5615cdfa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:26.401925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d521db7-7747-497c-9522-d4e5615cdfa9 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:26.429785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:26.516596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94cfc557-f5ba-471c-a4f5-3825542a30af service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:26.517194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cfc557-f5ba-471c-a4f5-3825542a30af service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:26.517472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cfc557-f5ba-471c-a4f5-3825542a30af service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:26.517742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94cfc557-f5ba-471c-a4f5-3825542a30af service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:26.519448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-94cfc557-f5ba-471c-a4f5-3825542a30af service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:26.607128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cfc557-f5ba-471c-a4f5-3825542a30af service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:26.608705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94cfc557-f5ba-471c-a4f5-3825542a30af service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:26.627335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 544/1117] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:28.691317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e83534-5a6f-4d7a-95be-e1a70b7af85b None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:28.691764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e83534-5a6f-4d7a-95be-e1a70b7af85b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:28.691865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e83534-5a6f-4d7a-95be-e1a70b7af85b None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:28.693190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 574/1118] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:28.698695 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3657af6-e83c-4eec-a817-f7a8184ca194 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:28.699162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3657af6-e83c-4eec-a817-f7a8184ca194 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:28.699265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3657af6-e83c-4eec-a817-f7a8184ca194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:28.739942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:28.763947 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c3657af6-e83c-4eec-a817-f7a8184ca194 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:28.765150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c3657af6-e83c-4eec-a817-f7a8184ca194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:28.860372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 545/1119] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:29.145922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e83534-5a6f-4d7a-95be-e1a70b7af85b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:29.147506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-08e83534-5a6f-4d7a-95be-e1a70b7af85b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:29.248868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e83534-5a6f-4d7a-95be-e1a70b7af85b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:29.280044 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:29.320411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f2e029-543f-46af-9c85-bf0806743ac6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:29.320999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f2e029-543f-46af-9c85-bf0806743ac6 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:29.321270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f2e029-543f-46af-9c85-bf0806743ac6 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:29.321536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f2e029-543f-46af-9c85-bf0806743ac6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:29.323130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-61f2e029-543f-46af-9c85-bf0806743ac6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:29.429192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f2e029-543f-46af-9c85-bf0806743ac6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:29.430629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f2e029-543f-46af-9c85-bf0806743ac6 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:29.448789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 575/1120] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:31.734762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418c8a61-e6d6-40ef-8f32-d0836331d510 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:31.735358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418c8a61-e6d6-40ef-8f32-d0836331d510 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:31.735358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-418c8a61-e6d6-40ef-8f32-d0836331d510 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:31.736695 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 546/1121] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:31.741815 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8f4e35-2fe2-435e-976e-dd257b56d481 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:31.742227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8f4e35-2fe2-435e-976e-dd257b56d481 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:31.742379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8f4e35-2fe2-435e-976e-dd257b56d481 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:31.809480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ef8f4e35-2fe2-435e-976e-dd257b56d481 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:31.810686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ef8f4e35-2fe2-435e-976e-dd257b56d481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:31.823203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:31.906094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 576/1122] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:32.193692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418c8a61-e6d6-40ef-8f32-d0836331d510 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:32.195308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-418c8a61-e6d6-40ef-8f32-d0836331d510 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:32.297894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-418c8a61-e6d6-40ef-8f32-d0836331d510 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:32.310914 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:32.370892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9d561b-4e09-4220-b603-3e941e1657fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:32.374041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9d561b-4e09-4220-b603-3e941e1657fb service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:32.374041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9d561b-4e09-4220-b603-3e941e1657fb service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:32.374041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9d561b-4e09-4220-b603-3e941e1657fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:32.374041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4e9d561b-4e09-4220-b603-3e941e1657fb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:32.486204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9d561b-4e09-4220-b603-3e941e1657fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:32.487664 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9d561b-4e09-4220-b603-3e941e1657fb service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:32.505767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 547/1123] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:34.996006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cf856-e7a1-438d-932e-56f0f4c3fcc3 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:34.996558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cf856-e7a1-438d-932e-56f0f4c3fcc3 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:34.996558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146cf856-e7a1-438d-932e-56f0f4c3fcc3 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:34.997881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 577/1124] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:35.003869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc30f29a-c8e3-4abe-824f-6cb54dab3547 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:35.004279 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc30f29a-c8e3-4abe-824f-6cb54dab3547 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:35.004442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc30f29a-c8e3-4abe-824f-6cb54dab3547 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:35.052104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:35.070554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-bc30f29a-c8e3-4abe-824f-6cb54dab3547 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:35.072868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-bc30f29a-c8e3-4abe-824f-6cb54dab3547 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:35.167593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 548/1125] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:35.446692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146cf856-e7a1-438d-932e-56f0f4c3fcc3 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:35.448273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-146cf856-e7a1-438d-932e-56f0f4c3fcc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:35.551191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146cf856-e7a1-438d-932e-56f0f4c3fcc3 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:35.590814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:35.624414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf615a8-c68c-4886-8198-cb5c14cdf50f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:35.625053 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf615a8-c68c-4886-8198-cb5c14cdf50f service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:35.625327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf615a8-c68c-4886-8198-cb5c14cdf50f service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:35.625594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf615a8-c68c-4886-8198-cb5c14cdf50f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:35.627194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3cf615a8-c68c-4886-8198-cb5c14cdf50f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:35.732814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf615a8-c68c-4886-8198-cb5c14cdf50f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:35.734232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf615a8-c68c-4886-8198-cb5c14cdf50f service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:35.752499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 578/1126] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:38.008953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a97a55-4b7d-417f-94bc-650e808c3f71 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:38.009515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a97a55-4b7d-417f-94bc-650e808c3f71 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:38.009515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a97a55-4b7d-417f-94bc-650e808c3f71 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:38.010850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 549/1127] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:38.016835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88330185-e5f0-4795-88f6-267ea12f12db None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:38.017284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88330185-e5f0-4795-88f6-267ea12f12db None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:38.017392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88330185-e5f0-4795-88f6-267ea12f12db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:38.081961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-88330185-e5f0-4795-88f6-267ea12f12db None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:38.083187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-88330185-e5f0-4795-88f6-267ea12f12db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:38.103082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:38.183881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 579/1128] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:38.475067 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a97a55-4b7d-417f-94bc-650e808c3f71 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:38.476670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a9a97a55-4b7d-417f-94bc-650e808c3f71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:38.579484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a97a55-4b7d-417f-94bc-650e808c3f71 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:38.599194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:38.650918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebd7033-caf9-4502-bef7-31d82ca8613b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:38.651543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd7033-caf9-4502-bef7-31d82ca8613b service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:38.651827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd7033-caf9-4502-bef7-31d82ca8613b service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:38.652099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebd7033-caf9-4502-bef7-31d82ca8613b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:38.653689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7ebd7033-caf9-4502-bef7-31d82ca8613b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:38.760108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebd7033-caf9-4502-bef7-31d82ca8613b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:38.761475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebd7033-caf9-4502-bef7-31d82ca8613b service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:38.779029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 550/1129] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:40.978861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4280c12-0381-4e38-a49a-a4527298f2f2 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:40.979436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4280c12-0381-4e38-a49a-a4527298f2f2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:40.979436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4280c12-0381-4e38-a49a-a4527298f2f2 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:40.980726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 580/1130] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:40.985955 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e00ca6-97a0-4971-bf72-520e73093314 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:40.986403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e00ca6-97a0-4971-bf72-520e73093314 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:40.986560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e00ca6-97a0-4971-bf72-520e73093314 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:41.049523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:41.052369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-d0e00ca6-97a0-4971-bf72-520e73093314 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:41.053612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d0e00ca6-97a0-4971-bf72-520e73093314 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:41.148707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 551/1131] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:41.433432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4280c12-0381-4e38-a49a-a4527298f2f2 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:41.434971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d4280c12-0381-4e38-a49a-a4527298f2f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:41.536917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4280c12-0381-4e38-a49a-a4527298f2f2 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:41.585534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:41.609722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51e5360-5f8f-4339-8556-8e2304c878ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:41.610315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51e5360-5f8f-4339-8556-8e2304c878ab service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:41.610601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51e5360-5f8f-4339-8556-8e2304c878ab service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:41.611118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51e5360-5f8f-4339-8556-8e2304c878ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:41.613193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d51e5360-5f8f-4339-8556-8e2304c878ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:41.717100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51e5360-5f8f-4339-8556-8e2304c878ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:41.719062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51e5360-5f8f-4339-8556-8e2304c878ab service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:41.737193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 581/1132] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:43.944948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace3dc93-0e57-4269-82a0-55825f612eac None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:43.945475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace3dc93-0e57-4269-82a0-55825f612eac None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:43.945475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace3dc93-0e57-4269-82a0-55825f612eac None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:43.946840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 552/1133] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:43.952328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bf5cf8-373f-4682-910c-6d9ac21043b3 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:43.953252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bf5cf8-373f-4682-910c-6d9ac21043b3 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:43.953406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bf5cf8-373f-4682-910c-6d9ac21043b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:44.019240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-41bf5cf8-373f-4682-910c-6d9ac21043b3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:44.020452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-41bf5cf8-373f-4682-910c-6d9ac21043b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:44.049178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:44.117154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 582/1134] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:44.403037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace3dc93-0e57-4269-82a0-55825f612eac None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:44.404599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ace3dc93-0e57-4269-82a0-55825f612eac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:44.511108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace3dc93-0e57-4269-82a0-55825f612eac None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:44.538806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:44.584549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d389fda7-d6c9-446a-b842-0066d0971911 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:44.585147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d389fda7-d6c9-446a-b842-0066d0971911 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:44.585423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d389fda7-d6c9-446a-b842-0066d0971911 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:44.585696 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d389fda7-d6c9-446a-b842-0066d0971911 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:44.587304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d389fda7-d6c9-446a-b842-0066d0971911 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:44.693559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d389fda7-d6c9-446a-b842-0066d0971911 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:44.695028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d389fda7-d6c9-446a-b842-0066d0971911 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:44.713818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 553/1135] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:46.899892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a58c085-fbdb-46cc-bb67-d497cd4579dd None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:46.900429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a58c085-fbdb-46cc-bb67-d497cd4579dd None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:46.900429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a58c085-fbdb-46cc-bb67-d497cd4579dd None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:46.901786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 583/1136] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:46.907139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f99ad-c562-4c2c-a3ef-c7751748a91b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:46.907542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f99ad-c562-4c2c-a3ef-c7751748a91b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:46.907684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f99ad-c562-4c2c-a3ef-c7751748a91b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:46.972750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-209f99ad-c562-4c2c-a3ef-c7751748a91b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:46.973873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-209f99ad-c562-4c2c-a3ef-c7751748a91b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:46.981980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:47.068103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 554/1137] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:54:47.358111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a58c085-fbdb-46cc-bb67-d497cd4579dd None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:47.360235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3a58c085-fbdb-46cc-bb67-d497cd4579dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:47.461974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a58c085-fbdb-46cc-bb67-d497cd4579dd None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:47.517093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:47.534909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc23879e-8752-4edf-bca1-ff6e75f894c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:47.535535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23879e-8752-4edf-bca1-ff6e75f894c5 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:47.535836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23879e-8752-4edf-bca1-ff6e75f894c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:47.536103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23879e-8752-4edf-bca1-ff6e75f894c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:47.537677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fc23879e-8752-4edf-bca1-ff6e75f894c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:47.644618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc23879e-8752-4edf-bca1-ff6e75f894c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:47.646093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc23879e-8752-4edf-bca1-ff6e75f894c5 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:47.664135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 584/1138] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:49.837471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561753e-8484-4f5d-b897-6489f54f3500 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:49.837988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561753e-8484-4f5d-b897-6489f54f3500 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:49.838101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2561753e-8484-4f5d-b897-6489f54f3500 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:49.839469 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 555/1139] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:49.844789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c2614d-8419-4f61-af1f-48eb1b4f5c68 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:49.845168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c2614d-8419-4f61-af1f-48eb1b4f5c68 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:49.845341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c2614d-8419-4f61-af1f-48eb1b4f5c68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:49.911600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-17c2614d-8419-4f61-af1f-48eb1b4f5c68 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:49.912846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-17c2614d-8419-4f61-af1f-48eb1b4f5c68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:49.952265 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:50.009726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 585/1140] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:50.300051 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2561753e-8484-4f5d-b897-6489f54f3500 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:50.301584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2561753e-8484-4f5d-b897-6489f54f3500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:50.404513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2561753e-8484-4f5d-b897-6489f54f3500 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:50.442827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:50.477666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25404601-2324-45ca-b34a-874476db26dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:50.478265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25404601-2324-45ca-b34a-874476db26dc service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:50.478540 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25404601-2324-45ca-b34a-874476db26dc service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:50.478806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25404601-2324-45ca-b34a-874476db26dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:50.480429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-25404601-2324-45ca-b34a-874476db26dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:50.591032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25404601-2324-45ca-b34a-874476db26dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:50.592453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25404601-2324-45ca-b34a-874476db26dc service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:50.611010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 556/1141] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:52.747407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eba86e1-d008-493c-a900-a7780adf0744 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:52.747893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eba86e1-d008-493c-a900-a7780adf0744 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:52.748007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eba86e1-d008-493c-a900-a7780adf0744 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:52.749303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 586/1142] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:52.754574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd5363-5f9a-4582-9965-86bb91ac42e6 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:52.755034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd5363-5f9a-4582-9965-86bb91ac42e6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:52.755132 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62cd5363-5f9a-4582-9965-86bb91ac42e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:52.820193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-62cd5363-5f9a-4582-9965-86bb91ac42e6 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:52.821385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-62cd5363-5f9a-4582-9965-86bb91ac42e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:52.839388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:52.918608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 557/1143] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:54:53.201223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eba86e1-d008-493c-a900-a7780adf0744 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:53.203417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9eba86e1-d008-493c-a900-a7780adf0744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:53.306448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eba86e1-d008-493c-a900-a7780adf0744 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:53.371983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:53.378938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8315b3b4-ed86-4d0f-a32a-55f2eac3210a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:53.383472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8315b3b4-ed86-4d0f-a32a-55f2eac3210a service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:53.383472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8315b3b4-ed86-4d0f-a32a-55f2eac3210a service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:53.383472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8315b3b4-ed86-4d0f-a32a-55f2eac3210a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:53.383472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8315b3b4-ed86-4d0f-a32a-55f2eac3210a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:53.488143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8315b3b4-ed86-4d0f-a32a-55f2eac3210a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:53.489528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8315b3b4-ed86-4d0f-a32a-55f2eac3210a service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:53.507814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 587/1144] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:54:55.670644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b253012-da4d-455b-a2b9-0ea02007b760 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:55.671275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b253012-da4d-455b-a2b9-0ea02007b760 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:55.671275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b253012-da4d-455b-a2b9-0ea02007b760 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:55.672619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 558/1145] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 21:54:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:54:55.678031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2996dadd-cc8a-4eee-978e-5f8b863bc85f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:55.678470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2996dadd-cc8a-4eee-978e-5f8b863bc85f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:55.678575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2996dadd-cc8a-4eee-978e-5f8b863bc85f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:55.743000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2996dadd-cc8a-4eee-978e-5f8b863bc85f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:54:55.744203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2996dadd-cc8a-4eee-978e-5f8b863bc85f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:55.792127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:55.839936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 588/1146] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 21:54:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:54:56.120762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b253012-da4d-455b-a2b9-0ea02007b760 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:54:56.122291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4b253012-da4d-455b-a2b9-0ea02007b760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:56.225152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b253012-da4d-455b-a2b9-0ea02007b760 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:54:56.271699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:54:56.302252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460458e7-740c-4280-8e9d-757b79da99ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:54:56.302861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460458e7-740c-4280-8e9d-757b79da99ff service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:54:56.303172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460458e7-740c-4280-8e9d-757b79da99ff service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:54:56.303444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460458e7-740c-4280-8e9d-757b79da99ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:54:56.305082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-460458e7-740c-4280-8e9d-757b79da99ff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:54:56.411737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460458e7-740c-4280-8e9d-757b79da99ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:54:56.413099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460458e7-740c-4280-8e9d-757b79da99ff service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:54:56.431303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 559/1147] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:54:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:55:05.838396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=22 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:06.429040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=66 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:16.244764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22ce592-1612-480d-bc27-946d76510ff8 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:16.245343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22ce592-1612-480d-bc27-946d76510ff8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:16.245343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22ce592-1612-480d-bc27-946d76510ff8 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:16.246706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 589/1148] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 21:55:16.252232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea6441-5928-48d2-9b92-cb0a4b275635 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:16.252661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea6441-5928-48d2-9b92-cb0a4b275635 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:16.252768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ea6441-5928-48d2-9b92-cb0a4b275635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:16.319260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-87ea6441-5928-48d2-9b92-cb0a4b275635 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:55:16.320452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-87ea6441-5928-48d2-9b92-cb0a4b275635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:16.415965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 560/1149] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:55:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:55:16.714167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b844dd-0cb0-416f-bf79-4c9aabee324f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:16.714447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b844dd-0cb0-416f-bf79-4c9aabee324f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:16.714723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b844dd-0cb0-416f-bf79-4c9aabee324f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:16.716097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 590/1150] 10.222.0.20 () {54 vars in 962 bytes} [Wed Oct 19 21:55:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:55:16.721849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d1ba3-d81c-4a51-9703-b5f4161c4d42 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:16.722138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d1ba3-d81c-4a51-9703-b5f4161c4d42 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:16.722414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d1ba3-d81c-4a51-9703-b5f4161c4d42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:16.799490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-983d1ba3-d81c-4a51-9703-b5f4161c4d42 None None] MFA Rules not processed for user `9ebc359d9bf54edcb9634afc31cf188a`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:55:16.800688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-983d1ba3-d81c-4a51-9703-b5f4161c4d42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:16.820764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:16.875560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 561/1151] 10.222.0.20 () {58 vars in 1059 bytes} [Wed Oct 19 21:55:16 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:55:16.883427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b844dd-0cb0-416f-bf79-4c9aabee324f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:55:16.884950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-16b844dd-0cb0-416f-bf79-4c9aabee324f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:16.987665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b844dd-0cb0-416f-bf79-4c9aabee324f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:55:17.059471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7e20cb-1201-428b-adc4-d4508952e076 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:55:17.060074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7e20cb-1201-428b-adc4-d4508952e076 service glance] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:17.060352 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7e20cb-1201-428b-adc4-d4508952e076 service glance] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:17.060622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7e20cb-1201-428b-adc4-d4508952e076 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:17.062780 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cf7e20cb-1201-428b-adc4-d4508952e076 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:17.115088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:17.168885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7e20cb-1201-428b-adc4-d4508952e076 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:55:17.170303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7e20cb-1201-428b-adc4-d4508952e076 service glance] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:55:17.189610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 591/1152] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:55:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:55:19.400096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af74af5-be35-42b7-8a7f-36d7f687f97f None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:19.400627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af74af5-be35-42b7-8a7f-36d7f687f97f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:19.400627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af74af5-be35-42b7-8a7f-36d7f687f97f None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:19.401910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 562/1153] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:55:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:55:19.407621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac2536-0c00-43ed-8421-210a2461a9de None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:19.408061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac2536-0c00-43ed-8421-210a2461a9de None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:19.408166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbac2536-0c00-43ed-8421-210a2461a9de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:19.474237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-fbac2536-0c00-43ed-8421-210a2461a9de None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:55:19.485773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fbac2536-0c00-43ed-8421-210a2461a9de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:19.571456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 592/1154] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:55:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:55:19.845392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af74af5-be35-42b7-8a7f-36d7f687f97f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:55:19.846946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4af74af5-be35-42b7-8a7f-36d7f687f97f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:19.972158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af74af5-be35-42b7-8a7f-36d7f687f97f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:55:20.020512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:20.064650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5a540-5c5f-46f6-9009-63eededa1f1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:55:20.065176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5a540-5c5f-46f6-9009-63eededa1f1e admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:20.065449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5a540-5c5f-46f6-9009-63eededa1f1e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:20.065715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5a540-5c5f-46f6-9009-63eededa1f1e admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:20.066381 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5a540-5c5f-46f6-9009-63eededa1f1e admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:55:20.067582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-f9f5a540-5c5f-46f6-9009-63eededa1f1e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:55:20.067582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 21:55:20.067582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 21:55:20.068713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5a540-5c5f-46f6-9009-63eededa1f1e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:55:20.087673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 563/1155] 10.222.0.20 () {56 vars in 1108 bytes} [Wed Oct 19 21:55:19 2022] GET /identity/v3/projects => generated 2553 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 21:55:22.237551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e37f6d0-b6ed-4934-b355-57cec2bb2b5e None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:22.238109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e37f6d0-b6ed-4934-b355-57cec2bb2b5e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:22.238109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e37f6d0-b6ed-4934-b355-57cec2bb2b5e None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:22.239461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 593/1156] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:55:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:55:22.244634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6e75d-22ff-40ab-9687-be75ef73116c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:22.245082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6e75d-22ff-40ab-9687-be75ef73116c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:22.245181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e6e75d-22ff-40ab-9687-be75ef73116c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:22.310105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-61e6e75d-22ff-40ab-9687-be75ef73116c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:55:22.311335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-61e6e75d-22ff-40ab-9687-be75ef73116c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:22.407477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 564/1157] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:55:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:55:22.707712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e37f6d0-b6ed-4934-b355-57cec2bb2b5e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:55:22.709308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5e37f6d0-b6ed-4934-b355-57cec2bb2b5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:22.719624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:22.858659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e37f6d0-b6ed-4934-b355-57cec2bb2b5e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:55:22.937191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:22.970023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8dcefcf-97ec-4851-ab7b-ad07aebf6e83 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:55:22.970559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcefcf-97ec-4851-ab7b-ad07aebf6e83 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:22.970830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcefcf-97ec-4851-ab7b-ad07aebf6e83 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:22.971121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8dcefcf-97ec-4851-ab7b-ad07aebf6e83 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:22.972705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f8dcefcf-97ec-4851-ab7b-ad07aebf6e83 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:23.058884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dcefcf-97ec-4851-ab7b-ad07aebf6e83 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:55:23.060496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8dcefcf-97ec-4851-ab7b-ad07aebf6e83 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:55:23.079246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 594/1158] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:55:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:55:25.393306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9282d-c734-4d9f-b11a-199d3431938c None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:25.393911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9282d-c734-4d9f-b11a-199d3431938c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:25.393911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d9282d-c734-4d9f-b11a-199d3431938c None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:25.395228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 565/1159] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:55:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:55:25.400333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab425253-8a8a-48f4-97e5-56ed39df042f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:25.400788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab425253-8a8a-48f4-97e5-56ed39df042f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:25.400898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab425253-8a8a-48f4-97e5-56ed39df042f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:25.465963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ab425253-8a8a-48f4-97e5-56ed39df042f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:55:25.467224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ab425253-8a8a-48f4-97e5-56ed39df042f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:25.513127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:25.563637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 595/1160] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:55:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 21:55:25.846529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9282d-c734-4d9f-b11a-199d3431938c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:55:25.848172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-44d9282d-c734-4d9f-b11a-199d3431938c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:25.871236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:25.948676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d9282d-c734-4d9f-b11a-199d3431938c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:55:26.020370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e26af-778b-4383-8c8b-0f3a0edd1a35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:55:26.020967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e26af-778b-4383-8c8b-0f3a0edd1a35 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:26.021242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e26af-778b-4383-8c8b-0f3a0edd1a35 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:26.021508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e26af-778b-4383-8c8b-0f3a0edd1a35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:26.023111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-736e26af-778b-4383-8c8b-0f3a0edd1a35 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:26.094674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:26.128218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736e26af-778b-4383-8c8b-0f3a0edd1a35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:55:26.129648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736e26af-778b-4383-8c8b-0f3a0edd1a35 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:55:26.147651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 566/1161] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:55:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:55:28.331500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d098ed-c80e-4a52-a9fc-0a8af479763f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:28.332515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d098ed-c80e-4a52-a9fc-0a8af479763f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:28.332515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d098ed-c80e-4a52-a9fc-0a8af479763f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:28.333506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 596/1162] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:55:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:55:28.339160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8055f7-731d-41d8-b375-fd0630aaab72 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:28.339592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8055f7-731d-41d8-b375-fd0630aaab72 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:28.339710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8055f7-731d-41d8-b375-fd0630aaab72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:28.405967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0c8055f7-731d-41d8-b375-fd0630aaab72 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:55:28.407196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0c8055f7-731d-41d8-b375-fd0630aaab72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:28.504614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 567/1163] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:55:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:55:28.796407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d098ed-c80e-4a52-a9fc-0a8af479763f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:55:28.797944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-24d098ed-c80e-4a52-a9fc-0a8af479763f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:28.905401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d098ed-c80e-4a52-a9fc-0a8af479763f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:55:28.975799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:28.977193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d47ada-2503-496d-8aaa-0d7fe7e5bba4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:55:28.977782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d47ada-2503-496d-8aaa-0d7fe7e5bba4 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:28.978056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d47ada-2503-496d-8aaa-0d7fe7e5bba4 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:28.978324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d47ada-2503-496d-8aaa-0d7fe7e5bba4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:28.979932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-52d47ada-2503-496d-8aaa-0d7fe7e5bba4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:29.088808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d47ada-2503-496d-8aaa-0d7fe7e5bba4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:55:29.090269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d47ada-2503-496d-8aaa-0d7fe7e5bba4 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:55:29.108660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 597/1164] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:55:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:55:31.337862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae0d44-1255-4e69-a178-55fe39d351ce None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:31.338455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae0d44-1255-4e69-a178-55fe39d351ce None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:31.338455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ae0d44-1255-4e69-a178-55fe39d351ce None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:31.339765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 568/1165] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:55:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:55:31.345127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab7169-8494-4bc4-b73d-71be8ed403ab None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:31.345575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab7169-8494-4bc4-b73d-71be8ed403ab None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:31.345707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ab7169-8494-4bc4-b73d-71be8ed403ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:31.409927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-89ab7169-8494-4bc4-b73d-71be8ed403ab None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:55:31.411154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-89ab7169-8494-4bc4-b73d-71be8ed403ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:31.463201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:31.505966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 598/1166] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:55:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:55:31.797144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ae0d44-1255-4e69-a178-55fe39d351ce None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:55:31.798710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b8ae0d44-1255-4e69-a178-55fe39d351ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:31.834369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:31.899356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ae0d44-1255-4e69-a178-55fe39d351ce None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:55:31.971583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db20266-cf5b-48f5-aee2-fef8df4595a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:55:31.972175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db20266-cf5b-48f5-aee2-fef8df4595a1 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:31.972448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db20266-cf5b-48f5-aee2-fef8df4595a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:31.972714 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db20266-cf5b-48f5-aee2-fef8df4595a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:31.974274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9db20266-cf5b-48f5-aee2-fef8df4595a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:32.053061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:32.081217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db20266-cf5b-48f5-aee2-fef8df4595a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:55:32.082643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db20266-cf5b-48f5-aee2-fef8df4595a1 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:55:32.100650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 569/1167] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:55:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:55:34.565991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7b81a-8553-4c13-ad8d-037add78ae13 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:34.566565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7b81a-8553-4c13-ad8d-037add78ae13 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:34.566565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba7b81a-8553-4c13-ad8d-037add78ae13 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:34.567979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 599/1168] 10.222.0.20 () {54 vars in 927 bytes} [Wed Oct 19 21:55:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:55:34.573333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080fd6d6-3d8d-4270-8af8-6d0bd83896fd None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:34.573738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080fd6d6-3d8d-4270-8af8-6d0bd83896fd None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:34.573881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080fd6d6-3d8d-4270-8af8-6d0bd83896fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:34.638959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-080fd6d6-3d8d-4270-8af8-6d0bd83896fd None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:55:34.640216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-080fd6d6-3d8d-4270-8af8-6d0bd83896fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:34.739146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 570/1169] 10.222.0.20 () {58 vars in 1024 bytes} [Wed Oct 19 21:55:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:55:35.034398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba7b81a-8553-4c13-ad8d-037add78ae13 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:55:35.035973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8ba7b81a-8553-4c13-ad8d-037add78ae13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:35.141302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba7b81a-8553-4c13-ad8d-037add78ae13 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:55:35.214284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77f5613-c4dd-47ab-8b6e-d6274859a069 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:55:35.214876 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f5613-c4dd-47ab-8b6e-d6274859a069 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:55:35.215175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f5613-c4dd-47ab-8b6e-d6274859a069 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:55:35.215443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f5613-c4dd-47ab-8b6e-d6274859a069 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:55:35.217014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f77f5613-c4dd-47ab-8b6e-d6274859a069 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:55:35.225761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:35.325375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77f5613-c4dd-47ab-8b6e-d6274859a069 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:55:35.326746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77f5613-c4dd-47ab-8b6e-d6274859a069 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:55:35.344884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 600/1170] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:55:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:55:44.737606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=82 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:55:45.342502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=50 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:58:35.532139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661b4de7-e786-4463-9a3e-08b61bb07c19 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:35.532739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661b4de7-e786-4463-9a3e-08b61bb07c19 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:35.532739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661b4de7-e786-4463-9a3e-08b61bb07c19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:35.617895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-661b4de7-e786-4463-9a3e-08b61bb07c19 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:58:35.619147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-661b4de7-e786-4463-9a3e-08b61bb07c19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:58:35.728730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 571/1171] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 21:58:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:58:35.736516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185bf3c-7699-4272-8896-f117c422dc36 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:35.737050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185bf3c-7699-4272-8896-f117c422dc36 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:35.737050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8185bf3c-7699-4272-8896-f117c422dc36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:35.757348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-8185bf3c-7699-4272-8896-f117c422dc36 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:58:35.758548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8185bf3c-7699-4272-8896-f117c422dc36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:58:35.767940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 601/1172] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 21:58:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:58:35.772638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661b4de7-e786-4463-9a3e-08b61bb07c19 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:35.783956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661b4de7-e786-4463-9a3e-08b61bb07c19 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:35.789472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:35.790085 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:35.790368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:35.790786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:35.791522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 21:58:35.791827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:35.792864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:58:35.792864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 21:58:35.792864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 21:58:35.793749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:35.806431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 572/1173] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 21:58:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 21:58:35.816467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8185bf3c-7699-4272-8896-f117c422dc36 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:35.825919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8185bf3c-7699-4272-8896-f117c422dc36 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:35.833817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:35.835423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:35.835715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:35.835992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:35.836649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:35.837764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING py.warnings [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:58:35.837764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: warnings.warn(msg) Oct 19 21:58:35.837764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: Oct 19 21:58:35.838998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:35.948540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 602/1174] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 21:58:35 2022] POST /identity/v3/projects => generated 379 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 21:58:35.952823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:35.963306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de16783-b891-4587-bb8a-d5b05eb8fd84 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:36.010916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:58:36.018388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:36.018974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.022524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.022524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.022524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:36.022524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:58:36.022524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 21:58:36.022524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 21:58:36.022524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:36.107175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 573/1175] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 21:58:35 2022] POST /identity/v3/users => generated 290 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 21:58:36.111623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:36.121904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd565d-7b49-42b2-8c23-fdf62ee047f0 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:36.127151 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66fa9e6-8163-44fd-988d-e9db9a419235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:36.127745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66fa9e6-8163-44fd-988d-e9db9a419235 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.128019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66fa9e6-8163-44fd-988d-e9db9a419235 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.128290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66fa9e6-8163-44fd-988d-e9db9a419235 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.128924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66fa9e6-8163-44fd-988d-e9db9a419235 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:36.130541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66fa9e6-8163-44fd-988d-e9db9a419235 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:36.143546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 603/1176] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 21:58:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 21:58:36.150145 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:36.157247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d35284-579a-480f-9553-21ba13876fca admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:36.161845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0c4a07-d2bd-4059-9935-a16c6c3024a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:36.162436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0c4a07-d2bd-4059-9935-a16c6c3024a8 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.162717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0c4a07-d2bd-4059-9935-a16c6c3024a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.163028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0c4a07-d2bd-4059-9935-a16c6c3024a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.163679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0c4a07-d2bd-4059-9935-a16c6c3024a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:36.165278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0c4a07-d2bd-4059-9935-a16c6c3024a8 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:36.179149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 574/1177] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 21:58:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 21:58:36.183127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66fa9e6-8163-44fd-988d-e9db9a419235 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:36.193411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66fa9e6-8163-44fd-988d-e9db9a419235 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:36.197894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f58f764-33ee-42e1-990c-5cde5dd7e943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:36.198520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58f764-33ee-42e1-990c-5cde5dd7e943 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.198792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58f764-33ee-42e1-990c-5cde5dd7e943 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.199098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f58f764-33ee-42e1-990c-5cde5dd7e943 admin admin] PATH_INFO: `/v3/projects/93bf88d70aae4791a8215472cc1089b4/users/2308240447a44486869263af3a763729/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.213214 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f58f764-33ee-42e1-990c-5cde5dd7e943 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93bf88d70aae4791a8215472cc1089b4, user_id=2308240447a44486869263af3a763729, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:36.215923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f58f764-33ee-42e1-990c-5cde5dd7e943 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:36.278284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 604/1178] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 21:58:36 2022] PUT /identity/v3/projects/93bf88d70aae4791a8215472cc1089b4/users/2308240447a44486869263af3a763729/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:58:36.289218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceff39-c196-47fe-938c-3e32f0887542 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.289507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceff39-c196-47fe-938c-3e32f0887542 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.289784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bceff39-c196-47fe-938c-3e32f0887542 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.318387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-9bceff39-c196-47fe-938c-3e32f0887542 None None] MFA Rules not processed for user `2308240447a44486869263af3a763729`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:58:36.319630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9bceff39-c196-47fe-938c-3e32f0887542 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:58:36.381849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 575/1179] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 21:58:36 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:58:36.394707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36cc08-8007-46cb-9df9-f111c5a04694 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.395032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36cc08-8007-46cb-9df9-f111c5a04694 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.395319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36cc08-8007-46cb-9df9-f111c5a04694 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.396643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 605/1180] 10.222.0.20 () {54 vars in 963 bytes} [Wed Oct 19 21:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:58:36.401365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda06906-3da9-4538-bdd7-51b2aa74ab83 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.401648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda06906-3da9-4538-bdd7-51b2aa74ab83 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.401942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda06906-3da9-4538-bdd7-51b2aa74ab83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.459205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-fda06906-3da9-4538-bdd7-51b2aa74ab83 None None] MFA Rules not processed for user `c065675e9a2f44f59168b9d574e289d5`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:58:36.460383 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fda06906-3da9-4538-bdd7-51b2aa74ab83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:58:36.534379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 576/1181] 10.222.0.20 () {58 vars in 1060 bytes} [Wed Oct 19 21:58:36 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:58:36.542157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da36cc08-8007-46cb-9df9-f111c5a04694 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:36.553443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da36cc08-8007-46cb-9df9-f111c5a04694 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:36.556964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7076e-9b57-4a8f-baab-da0ac49e9250 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:36.557548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7076e-9b57-4a8f-baab-da0ac49e9250 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.557827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7076e-9b57-4a8f-baab-da0ac49e9250 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.558098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4be7076e-9b57-4a8f-baab-da0ac49e9250 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.566321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be7076e-9b57-4a8f-baab-da0ac49e9250 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:36.567767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4be7076e-9b57-4a8f-baab-da0ac49e9250 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:36.576421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 606/1182] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:58:36 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 21:58:36.621088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22db76-b437-473d-9285-af0500fc2c0d None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.621369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22db76-b437-473d-9285-af0500fc2c0d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.621639 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22db76-b437-473d-9285-af0500fc2c0d None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.622884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 577/1183] 10.222.0.20 () {52 vars in 830 bytes} [Wed Oct 19 21:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 21:58:36.626457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7076e-9b57-4a8f-baab-da0ac49e9250 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:36.686489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4be7076e-9b57-4a8f-baab-da0ac49e9250 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:36.691075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:36.691689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:36.691961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:36.692229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] PATH_INFO: `/v3/users/2308240447a44486869263af3a763729` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:36.693320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] RBAC: Authorizing `identity:delete_user(user_id=2308240447a44486869263af3a763729)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:36.694322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING py.warnings [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:58:36.694322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: warnings.warn(msg) Oct 19 21:58:36.694322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: Oct 19 21:58:36.695576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:36.894437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2308240447a44486869263af3a763729'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:36.895231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2308240447a44486869263af3a763729'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:36.920227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2308240447a44486869263af3a763729'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:36.934113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2308240447a44486869263af3a763729'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:36.934829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2308240447a44486869263af3a763729'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:36.938892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:58:37.031751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 607/1184] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 21:58:36 2022] DELETE /identity/v3/users/2308240447a44486869263af3a763729 => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:58:37.036891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c8516-81a2-453c-929a-11962756999a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:37.037178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c8516-81a2-453c-929a-11962756999a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:37.037453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c8516-81a2-453c-929a-11962756999a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:37.056428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f49c8516-81a2-453c-929a-11962756999a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 21:58:37.057612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f49c8516-81a2-453c-929a-11962756999a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:58:37.066715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:58:37.114667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 578/1185] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 21:58:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 21:58:37.124423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:37.125879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:58:37.202286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0efc9cf-5ef7-479e-aca9-d04b976462ae admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:37.206434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c884e8e-5248-4cd4-98bd-620c09f6a243 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:37.207052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c884e8e-5248-4cd4-98bd-620c09f6a243 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:37.207329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c884e8e-5248-4cd4-98bd-620c09f6a243 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:37.207599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c884e8e-5248-4cd4-98bd-620c09f6a243 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:37.216279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c884e8e-5248-4cd4-98bd-620c09f6a243 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:37.217688 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c884e8e-5248-4cd4-98bd-620c09f6a243 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:37.226908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 608/1186] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 21:58:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 21:58:37.676239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c8516-81a2-453c-929a-11962756999a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 21:58:37.677669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f49c8516-81a2-453c-929a-11962756999a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 21:58:37.689997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c8516-81a2-453c-929a-11962756999a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 21:58:37.695629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 21:58:37.696473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 21:58:37.696777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 21:58:37.697050 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] PATH_INFO: `/v3/projects/93bf88d70aae4791a8215472cc1089b4` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 21:58:37.698222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] RBAC: Authorizing `identity:delete_project(project_id=93bf88d70aae4791a8215472cc1089b4)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 21:58:37.699442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 21:58:37.699442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 21:58:37.699442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 21:58:37.701561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 21:58:37.819042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93bf88d70aae4791a8215472cc1089b4', 'request_id': 'req-53621eb6-c791-4c13-a610-e428c0ecbbda'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:37.884736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93bf88d70aae4791a8215472cc1089b4', 'request_id': 'req-53621eb6-c791-4c13-a610-e428c0ecbbda'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:37.891186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93bf88d70aae4791a8215472cc1089b4', 'request_id': 'req-53621eb6-c791-4c13-a610-e428c0ecbbda'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:37.910393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] The token cache is being invalidate because project 93bf88d70aae4791a8215472cc1089b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 21:58:37.910683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:37.911464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-53621eb6-c791-4c13-a610-e428c0ecbbda admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 21:58:37.912962 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 579/1187] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 21:58:37 2022] DELETE /identity/v3/projects/93bf88d70aae4791a8215472cc1089b4 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 21:58:47.222827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=1,keystone:SELECT=37 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 21:58:47.906857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=47,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:00:02.189308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd49958-c4e4-442d-9021-9865d05dc959 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:02.189877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd49958-c4e4-442d-9021-9865d05dc959 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:02.189877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd49958-c4e4-442d-9021-9865d05dc959 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:02.191168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 609/1188] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 22:00:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:02.196032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d1d23-0824-419e-8afe-b0383882e5b0 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:02.196481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d1d23-0824-419e-8afe-b0383882e5b0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:02.196677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37d1d23-0824-419e-8afe-b0383882e5b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:02.216204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c37d1d23-0824-419e-8afe-b0383882e5b0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:02.219696 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c37d1d23-0824-419e-8afe-b0383882e5b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:02.280114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 580/1189] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 22:00:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:02.435803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd49958-c4e4-442d-9021-9865d05dc959 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:02.437537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ecd49958-c4e4-442d-9021-9865d05dc959 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:02.551880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd49958-c4e4-442d-9021-9865d05dc959 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:02.557372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ea677b-ad2e-4003-8b3e-3827e6ca3de4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:02.557971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea677b-ad2e-4003-8b3e-3827e6ca3de4 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:02.558250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea677b-ad2e-4003-8b3e-3827e6ca3de4 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:02.558523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ea677b-ad2e-4003-8b3e-3827e6ca3de4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:02.566388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ea677b-ad2e-4003-8b3e-3827e6ca3de4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:02.567998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ea677b-ad2e-4003-8b3e-3827e6ca3de4 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:02.577210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 610/1190] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:00:03.660942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6daa33-e820-4183-b516-eace1d9d6cf5 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:03.661473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6daa33-e820-4183-b516-eace1d9d6cf5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:03.661473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6daa33-e820-4183-b516-eace1d9d6cf5 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:03.662809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 581/1191] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 22:00:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:03.668828 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d918294e-d788-45fd-a52c-98bc2aa17dfa None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:03.669262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d918294e-d788-45fd-a52c-98bc2aa17dfa None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:03.669371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d918294e-d788-45fd-a52c-98bc2aa17dfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:03.689433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d918294e-d788-45fd-a52c-98bc2aa17dfa None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:03.690619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d918294e-d788-45fd-a52c-98bc2aa17dfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:03.698295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 611/1192] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 22:00:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:03.849652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6daa33-e820-4183-b516-eace1d9d6cf5 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:03.864078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6daa33-e820-4183-b516-eace1d9d6cf5 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:03.869621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9362e91-a030-4d84-b0d4-4f90cc0da09b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:03.870282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9362e91-a030-4d84-b0d4-4f90cc0da09b service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:03.870596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9362e91-a030-4d84-b0d4-4f90cc0da09b service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:03.870869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9362e91-a030-4d84-b0d4-4f90cc0da09b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:03.879081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9362e91-a030-4d84-b0d4-4f90cc0da09b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:03.880717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9362e91-a030-4d84-b0d4-4f90cc0da09b service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:03.890174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 582/1193] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:00:04.545439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a8cb45-f9c0-449f-80dc-a283d83ca8f0 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:04.545702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a8cb45-f9c0-449f-80dc-a283d83ca8f0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:04.545982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a8cb45-f9c0-449f-80dc-a283d83ca8f0 None None] PATH_INFO: `/v3` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:04.547301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 612/1194] 10.222.0.13 () {54 vars in 925 bytes} [Wed Oct 19 22:00:04 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:00:04.553967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c05e1-12f8-4d2f-ae91-cff95e49191c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:04.554255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c05e1-12f8-4d2f-ae91-cff95e49191c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:04.554528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852c05e1-12f8-4d2f-ae91-cff95e49191c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:04.602636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-852c05e1-12f8-4d2f-ae91-cff95e49191c None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:04.603926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-852c05e1-12f8-4d2f-ae91-cff95e49191c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:04.674218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 583/1195] 10.222.0.13 () {58 vars in 1015 bytes} [Wed Oct 19 22:00:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:08.528658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f8978-8ba5-4642-bfa1-f0aeeaef7f72 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:08.529067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f8978-8ba5-4642-bfa1-f0aeeaef7f72 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:08.529221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120f8978-8ba5-4642-bfa1-f0aeeaef7f72 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:08.530552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 613/1196] 10.222.0.20 () {54 vars in 961 bytes} [Wed Oct 19 22:00:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:08.536179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3fc93c-e2e8-464d-bf2a-3b05740a59c0 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:08.536570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3fc93c-e2e8-464d-bf2a-3b05740a59c0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:08.545526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3fc93c-e2e8-464d-bf2a-3b05740a59c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:08.556123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6d3fc93c-e2e8-464d-bf2a-3b05740a59c0 None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:08.557323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6d3fc93c-e2e8-464d-bf2a-3b05740a59c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:08.565922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 584/1197] 10.222.0.20 () {58 vars in 1058 bytes} [Wed Oct 19 22:00:08 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:00:08.573141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120f8978-8ba5-4642-bfa1-f0aeeaef7f72 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:08.586627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120f8978-8ba5-4642-bfa1-f0aeeaef7f72 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:08.591454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd03251-c6b4-407d-88b2-0223adc36960 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:08.592024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd03251-c6b4-407d-88b2-0223adc36960 service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:08.592299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd03251-c6b4-407d-88b2-0223adc36960 service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:08.592565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd03251-c6b4-407d-88b2-0223adc36960 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:08.601158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd03251-c6b4-407d-88b2-0223adc36960 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:08.602744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd03251-c6b4-407d-88b2-0223adc36960 service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:08.613341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 614/1198] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:08 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:00:15.365359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051016ad-831a-41e2-a3ec-0db7ab614c47 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:15.365952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051016ad-831a-41e2-a3ec-0db7ab614c47 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:15.365952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051016ad-831a-41e2-a3ec-0db7ab614c47 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:15.367296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 585/1199] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 22:00:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:15.373295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf43213-dc51-4f03-b112-6b5e49846013 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:15.373749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf43213-dc51-4f03-b112-6b5e49846013 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:15.373866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf43213-dc51-4f03-b112-6b5e49846013 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:15.393005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6bf43213-dc51-4f03-b112-6b5e49846013 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:15.394210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6bf43213-dc51-4f03-b112-6b5e49846013 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:15.402152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 615/1200] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 22:00:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:15.555943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051016ad-831a-41e2-a3ec-0db7ab614c47 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:15.570661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051016ad-831a-41e2-a3ec-0db7ab614c47 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:15.578506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac9100f9-73fe-4156-8d48-6968726bef4c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:15.578506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9100f9-73fe-4156-8d48-6968726bef4c service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:15.578506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9100f9-73fe-4156-8d48-6968726bef4c service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:15.578506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac9100f9-73fe-4156-8d48-6968726bef4c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:15.585522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9100f9-73fe-4156-8d48-6968726bef4c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:15.587133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac9100f9-73fe-4156-8d48-6968726bef4c service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:15.597210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 586/1201] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:00:16.668582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d03c84b-592d-4652-a4de-2062224a2e46 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:16.669111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d03c84b-592d-4652-a4de-2062224a2e46 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:16.669111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d03c84b-592d-4652-a4de-2062224a2e46 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:16.670395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 616/1202] 127.0.0.1 () {54 vars in 901 bytes} [Wed Oct 19 22:00:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:16.677291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e685b-bf39-47c5-af58-22acda7ee956 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:16.677715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e685b-bf39-47c5-af58-22acda7ee956 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:16.677843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340e685b-bf39-47c5-af58-22acda7ee956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:16.699080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-340e685b-bf39-47c5-af58-22acda7ee956 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:16.700276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-340e685b-bf39-47c5-af58-22acda7ee956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:16.708023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 587/1203] 127.0.0.1 () {58 vars in 998 bytes} [Wed Oct 19 22:00:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:16.857758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d03c84b-592d-4652-a4de-2062224a2e46 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:16.872521 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d03c84b-592d-4652-a4de-2062224a2e46 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:16.878062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b71e882-0094-4d87-bcf2-f401cd7c5e76 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:16.878629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71e882-0094-4d87-bcf2-f401cd7c5e76 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:16.878907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71e882-0094-4d87-bcf2-f401cd7c5e76 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:16.879210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b71e882-0094-4d87-bcf2-f401cd7c5e76 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:16.887222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b71e882-0094-4d87-bcf2-f401cd7c5e76 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:16.888787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b71e882-0094-4d87-bcf2-f401cd7c5e76 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:16.898030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 617/1204] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:00:20.682748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579575-cc17-4835-ba8e-ae7193563097 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:20.683327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579575-cc17-4835-ba8e-ae7193563097 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:20.683327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579575-cc17-4835-ba8e-ae7193563097 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:20.684663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 588/1205] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 22:00:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:20.690374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d03bc3-998e-4cce-b95b-4f847c1e44d4 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:20.690820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d03bc3-998e-4cce-b95b-4f847c1e44d4 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:20.690930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d03bc3-998e-4cce-b95b-4f847c1e44d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:20.710467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-72d03bc3-998e-4cce-b95b-4f847c1e44d4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:20.711701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-72d03bc3-998e-4cce-b95b-4f847c1e44d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:20.719435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 618/1206] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 22:00:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:21.005454 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd579575-cc17-4835-ba8e-ae7193563097 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:21.007241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fd579575-cc17-4835-ba8e-ae7193563097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:21.034564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:00:21.063586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd579575-cc17-4835-ba8e-ae7193563097 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:21.067763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db91ed44-8837-43b1-b34e-a1f90ca3c962 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:21.068363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db91ed44-8837-43b1-b34e-a1f90ca3c962 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:21.068636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db91ed44-8837-43b1-b34e-a1f90ca3c962 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:21.068907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db91ed44-8837-43b1-b34e-a1f90ca3c962 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:21.077947 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db91ed44-8837-43b1-b34e-a1f90ca3c962 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:21.079429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db91ed44-8837-43b1-b34e-a1f90ca3c962 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:21.089121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 589/1207] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:00:23.204195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901ee2d6-12bf-457c-892a-8ce75f8f35d1 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:23.204738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901ee2d6-12bf-457c-892a-8ce75f8f35d1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:23.204833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901ee2d6-12bf-457c-892a-8ce75f8f35d1 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:23.206630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 619/1208] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 22:00:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:23.211745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71957d54-131e-436a-bd60-fbe84a543f53 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:23.212170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71957d54-131e-436a-bd60-fbe84a543f53 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:23.212278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71957d54-131e-436a-bd60-fbe84a543f53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:23.231785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-71957d54-131e-436a-bd60-fbe84a543f53 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:23.232967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-71957d54-131e-436a-bd60-fbe84a543f53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:23.240658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 590/1209] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 22:00:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:23.525160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901ee2d6-12bf-457c-892a-8ce75f8f35d1 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:23.536562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901ee2d6-12bf-457c-892a-8ce75f8f35d1 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:23.540068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6262795f-3217-4706-87f8-c4e8ca0edcc9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:23.540651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6262795f-3217-4706-87f8-c4e8ca0edcc9 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:23.540926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6262795f-3217-4706-87f8-c4e8ca0edcc9 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:23.541192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6262795f-3217-4706-87f8-c4e8ca0edcc9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:23.550122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6262795f-3217-4706-87f8-c4e8ca0edcc9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:23.551574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6262795f-3217-4706-87f8-c4e8ca0edcc9 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:23.560668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 620/1210] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:00:27.085374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962de72e-0b53-4465-9544-fbd4f517b342 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:27.085902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962de72e-0b53-4465-9544-fbd4f517b342 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:27.085902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962de72e-0b53-4465-9544-fbd4f517b342 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:27.087268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 591/1211] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 22:00:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:27.092399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d25584a-6b5e-4336-9e95-cb811e033014 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:27.092793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d25584a-6b5e-4336-9e95-cb811e033014 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:27.092956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d25584a-6b5e-4336-9e95-cb811e033014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:27.112413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3d25584a-6b5e-4336-9e95-cb811e033014 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:27.113599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3d25584a-6b5e-4336-9e95-cb811e033014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:27.121314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 621/1212] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 22:00:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:27.421319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962de72e-0b53-4465-9544-fbd4f517b342 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:27.423048 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-962de72e-0b53-4465-9544-fbd4f517b342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:27.479761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962de72e-0b53-4465-9544-fbd4f517b342 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:27.483310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e472fdd-6bbc-4d27-b226-041201f7ef5e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:27.483900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e472fdd-6bbc-4d27-b226-041201f7ef5e service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:27.484196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e472fdd-6bbc-4d27-b226-041201f7ef5e service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:27.484465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e472fdd-6bbc-4d27-b226-041201f7ef5e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:27.493108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e472fdd-6bbc-4d27-b226-041201f7ef5e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:27.494525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e472fdd-6bbc-4d27-b226-041201f7ef5e service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:27.504263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 592/1213] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:00:29.648026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72270d2c-8066-4073-8e82-1c0fdcc06913 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:29.648563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72270d2c-8066-4073-8e82-1c0fdcc06913 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:29.648563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72270d2c-8066-4073-8e82-1c0fdcc06913 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:29.649907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 622/1214] 127.0.0.1 () {54 vars in 917 bytes} [Wed Oct 19 22:00:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:00:29.655067 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92332876-a7a8-46e1-841d-2ae5ca490cd0 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:29.655452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92332876-a7a8-46e1-841d-2ae5ca490cd0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:29.655629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92332876-a7a8-46e1-841d-2ae5ca490cd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:29.676255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-92332876-a7a8-46e1-841d-2ae5ca490cd0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:00:29.677452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-92332876-a7a8-46e1-841d-2ae5ca490cd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:29.685296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 593/1215] 127.0.0.1 () {58 vars in 1014 bytes} [Wed Oct 19 22:00:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:00:29.989720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72270d2c-8066-4073-8e82-1c0fdcc06913 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:00:29.991450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-72270d2c-8066-4073-8e82-1c0fdcc06913 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:00:29.999323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=77 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:00:30.005779 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72270d2c-8066-4073-8e82-1c0fdcc06913 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:00:30.009325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6be6281-aa4d-4aa6-a52d-2846f6ae93ee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:00:30.009914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be6281-aa4d-4aa6-a52d-2846f6ae93ee service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:00:30.010196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be6281-aa4d-4aa6-a52d-2846f6ae93ee service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:00:30.010467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6be6281-aa4d-4aa6-a52d-2846f6ae93ee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:00:30.019301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be6281-aa4d-4aa6-a52d-2846f6ae93ee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:00:30.020719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6be6281-aa4d-4aa6-a52d-2846f6ae93ee service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:00:30.030076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 623/1216] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:00:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:00:39.670572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=47 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:00:40.119569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:55.176913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a2ed1-7ecb-44bc-b8af-769e1558446a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.185652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a2ed1-7ecb-44bc-b8af-769e1558446a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.185652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a2ed1-7ecb-44bc-b8af-769e1558446a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.204678 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ad6a2ed1-7ecb-44bc-b8af-769e1558446a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.205960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ad6a2ed1-7ecb-44bc-b8af-769e1558446a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.219509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 594/1217] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.230524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee905c8-4692-4237-bdc2-c312a8b041eb None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.231179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee905c8-4692-4237-bdc2-c312a8b041eb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.231179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee905c8-4692-4237-bdc2-c312a8b041eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.258802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-bee905c8-4692-4237-bdc2-c312a8b041eb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.260212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-bee905c8-4692-4237-bdc2-c312a8b041eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.277894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 624/1218] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.283078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a2ed1-7ecb-44bc-b8af-769e1558446a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:55.293814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a2ed1-7ecb-44bc-b8af-769e1558446a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:55.307106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76884f2-dd50-4481-bd3b-77fe387f2272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:55.307741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76884f2-dd50-4481-bd3b-77fe387f2272 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.308018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76884f2-dd50-4481-bd3b-77fe387f2272 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.308292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76884f2-dd50-4481-bd3b-77fe387f2272 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.308961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76884f2-dd50-4481-bd3b-77fe387f2272 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:01:55.309266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76884f2-dd50-4481-bd3b-77fe387f2272 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:55.310881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76884f2-dd50-4481-bd3b-77fe387f2272 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:55.332603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 595/1219] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:01:55.341737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee905c8-4692-4237-bdc2-c312a8b041eb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:55.352045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee905c8-4692-4237-bdc2-c312a8b041eb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:55.362833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a126de3a-9310-4ec4-ae37-33a8ec33d110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:55.363520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a126de3a-9310-4ec4-ae37-33a8ec33d110 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.363797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a126de3a-9310-4ec4-ae37-33a8ec33d110 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.364122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a126de3a-9310-4ec4-ae37-33a8ec33d110 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.364771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a126de3a-9310-4ec4-ae37-33a8ec33d110 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:55.366720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a126de3a-9310-4ec4-ae37-33a8ec33d110 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:55.484303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55671bd3-7257-4d96-ba4e-a1dcefa96ffb None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.484715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55671bd3-7257-4d96-ba4e-a1dcefa96ffb None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.484851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55671bd3-7257-4d96-ba4e-a1dcefa96ffb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.501135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 625/1220] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/projects => generated 387 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:55.502463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf8020-ec52-4d56-89d5-bfa9accb6120 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.502749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf8020-ec52-4d56-89d5-bfa9accb6120 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.503045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf8020-ec52-4d56-89d5-bfa9accb6120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.520036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-55671bd3-7257-4d96-ba4e-a1dcefa96ffb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.521345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-55671bd3-7257-4d96-ba4e-a1dcefa96ffb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.527021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-4ebf8020-ec52-4d56-89d5-bfa9accb6120 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.528227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4ebf8020-ec52-4d56-89d5-bfa9accb6120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.621410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 626/1221] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.622270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebf8020-ec52-4d56-89d5-bfa9accb6120 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:55.633613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 596/1222] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:01:55.637188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d990fbfa-0ad0-4f23-b3dd-c2604201232c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.637487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d990fbfa-0ad0-4f23-b3dd-c2604201232c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.637764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d990fbfa-0ad0-4f23-b3dd-c2604201232c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.643107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebf8020-ec52-4d56-89d5-bfa9accb6120 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:55.652699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808488db-f578-45ce-a45f-31429fb8c028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:55.653316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808488db-f578-45ce-a45f-31429fb8c028 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.653591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808488db-f578-45ce-a45f-31429fb8c028 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.653860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808488db-f578-45ce-a45f-31429fb8c028 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.654497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808488db-f578-45ce-a45f-31429fb8c028 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:55.655548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING py.warnings [None req-808488db-f578-45ce-a45f-31429fb8c028 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 22:01:55.655548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: warnings.warn(msg) Oct 19 22:01:55.655548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: Oct 19 22:01:55.658821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808488db-f578-45ce-a45f-31429fb8c028 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:55.673101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-d990fbfa-0ad0-4f23-b3dd-c2604201232c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.674408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d990fbfa-0ad0-4f23-b3dd-c2604201232c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.687470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 597/1223] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.689091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb2a72e-acb1-4afc-a089-0d255870045c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.689375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb2a72e-acb1-4afc-a089-0d255870045c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.689654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb2a72e-acb1-4afc-a089-0d255870045c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.712784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-fcb2a72e-acb1-4afc-a089-0d255870045c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.713984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fcb2a72e-acb1-4afc-a089-0d255870045c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.722521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 598/1224] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.723967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00730dcf-ae2f-459f-b9a4-394ab58358b9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.724287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00730dcf-ae2f-459f-b9a4-394ab58358b9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.724560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00730dcf-ae2f-459f-b9a4-394ab58358b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.744350 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-00730dcf-ae2f-459f-b9a4-394ab58358b9 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.745468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-00730dcf-ae2f-459f-b9a4-394ab58358b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.753216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 599/1225] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.754485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d2288-4baa-4c4e-85ad-d4d203eab87a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.755441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d2288-4baa-4c4e-85ad-d4d203eab87a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.755718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525d2288-4baa-4c4e-85ad-d4d203eab87a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.774000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-525d2288-4baa-4c4e-85ad-d4d203eab87a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.775174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-525d2288-4baa-4c4e-85ad-d4d203eab87a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.782773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 600/1226] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.784062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0547316e-f9de-4093-8927-d4c80a121536 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.784345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0547316e-f9de-4093-8927-d4c80a121536 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.784619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0547316e-f9de-4093-8927-d4c80a121536 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.804493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 627/1227] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/users => generated 294 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:55.804493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb05ca5b-e59a-4b22-a423-ad16dd68192c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.804493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb05ca5b-e59a-4b22-a423-ad16dd68192c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.804493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb05ca5b-e59a-4b22-a423-ad16dd68192c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.806939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0547316e-f9de-4093-8927-d4c80a121536 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.808295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0547316e-f9de-4093-8927-d4c80a121536 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.816283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 601/1228] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.818945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001b478e-1ccf-4370-a8a3-7ef67336c6e9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.818945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001b478e-1ccf-4370-a8a3-7ef67336c6e9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.818945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001b478e-1ccf-4370-a8a3-7ef67336c6e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.822676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-fb05ca5b-e59a-4b22-a423-ad16dd68192c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.824033 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fb05ca5b-e59a-4b22-a423-ad16dd68192c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.832170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 628/1229] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.833372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5f2824-41ee-451a-bf0c-4bae511d7e57 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.833653 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5f2824-41ee-451a-bf0c-4bae511d7e57 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.833972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5f2824-41ee-451a-bf0c-4bae511d7e57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.838435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-001b478e-1ccf-4370-a8a3-7ef67336c6e9 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.839669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-001b478e-1ccf-4370-a8a3-7ef67336c6e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.848030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 602/1230] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.849701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d366ae-aef9-4636-8ac3-6d66dde6eadf None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.849980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d366ae-aef9-4636-8ac3-6d66dde6eadf None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.850251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d366ae-aef9-4636-8ac3-6d66dde6eadf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.853247 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-fe5f2824-41ee-451a-bf0c-4bae511d7e57 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.854420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fe5f2824-41ee-451a-bf0c-4bae511d7e57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.863456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 629/1231] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.865025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5f2824-41ee-451a-bf0c-4bae511d7e57 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:55.870240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-59d366ae-aef9-4636-8ac3-6d66dde6eadf None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.871475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-59d366ae-aef9-4636-8ac3-6d66dde6eadf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.874938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5f2824-41ee-451a-bf0c-4bae511d7e57 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:55.879451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:55.882965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 603/1232] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.883109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.883109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.883315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d366ae-aef9-4636-8ac3-6d66dde6eadf None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:55.883416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.883416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:01:55.883416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:55.883416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING py.warnings [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 19 22:01:55.883416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: warnings.warn(msg) Oct 19 22:01:55.883416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: Oct 19 22:01:55.883903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:55.890111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d366ae-aef9-4636-8ac3-6d66dde6eadf None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:55.894638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7336e92-661b-480a-aa19-dfbdba93b227 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:55.895247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7336e92-661b-480a-aa19-dfbdba93b227 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.895445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 630/1233] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:01:55.895589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7336e92-661b-480a-aa19-dfbdba93b227 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.895807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7336e92-661b-480a-aa19-dfbdba93b227 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.896299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:55.896477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7336e92-661b-480a-aa19-dfbdba93b227 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:01:55.896744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7336e92-661b-480a-aa19-dfbdba93b227 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:55.898551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7336e92-661b-480a-aa19-dfbdba93b227 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:55.910414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 604/1234] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:01:55.912146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f80ebe5-e4c6-4ffd-ae9e-bed4328feabd None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.912426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f80ebe5-e4c6-4ffd-ae9e-bed4328feabd None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.912697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f80ebe5-e4c6-4ffd-ae9e-bed4328feabd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.914422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95a2da5-5b8b-4101-8171-40ed0db6640e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:55.919766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d284af-b8ac-46c6-92cb-cb9dba30c691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:55.920365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d284af-b8ac-46c6-92cb-cb9dba30c691 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.920641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d284af-b8ac-46c6-92cb-cb9dba30c691 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.920911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d284af-b8ac-46c6-92cb-cb9dba30c691 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.921555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d284af-b8ac-46c6-92cb-cb9dba30c691 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:55.923163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d284af-b8ac-46c6-92cb-cb9dba30c691 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:55.932545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-4f80ebe5-e4c6-4ffd-ae9e-bed4328feabd None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.933765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4f80ebe5-e4c6-4ffd-ae9e-bed4328feabd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.934395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 631/1235] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:55.935239 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d284af-b8ac-46c6-92cb-cb9dba30c691 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:55.942123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 605/1236] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.943491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4f91b-03a3-4c28-8749-fb66f74e5035 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.943771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4f91b-03a3-4c28-8749-fb66f74e5035 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.944037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f4f91b-03a3-4c28-8749-fb66f74e5035 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.946610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d284af-b8ac-46c6-92cb-cb9dba30c691 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:55.951312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c339ec4-4f6a-4565-9741-f599eed939e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:55.951891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c339ec4-4f6a-4565-9741-f599eed939e3 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.952974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c339ec4-4f6a-4565-9741-f599eed939e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.952974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c339ec4-4f6a-4565-9741-f599eed939e3 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.953198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c339ec4-4f6a-4565-9741-f599eed939e3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:01:55.953353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c339ec4-4f6a-4565-9741-f599eed939e3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:55.954921 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c339ec4-4f6a-4565-9741-f599eed939e3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:55.964630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b4f4f91b-03a3-4c28-8749-fb66f74e5035 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.965810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b4f4f91b-03a3-4c28-8749-fb66f74e5035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.966330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 632/1237] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:01:55.967509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d1e12b-2c75-4f67-911e-69c4cf1b88af None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.967794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d1e12b-2c75-4f67-911e-69c4cf1b88af None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.968069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d1e12b-2c75-4f67-911e-69c4cf1b88af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.975449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 606/1238] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:55.976299 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f4f91b-03a3-4c28-8749-fb66f74e5035 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:55.985463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f4f91b-03a3-4c28-8749-fb66f74e5035 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:55.990537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:55.990710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-08d1e12b-2c75-4f67-911e-69c4cf1b88af None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:55.991258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:55.991531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:55.991797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:55.992438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:01:55.992545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-08d1e12b-2c75-4f67-911e-69c4cf1b88af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:55.992736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:55.994320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.000339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 633/1239] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:56.001041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d1e12b-2c75-4f67-911e-69c4cf1b88af None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.006184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 607/1240] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:01:56.007230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.012594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d1e12b-2c75-4f67-911e-69c4cf1b88af None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.018113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-234399ec-f95d-43f0-9f9a-a7457bbd0abf admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.018271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d19afa-2e37-42e1-82dc-5bddf293e0be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.018736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d19afa-2e37-42e1-82dc-5bddf293e0be admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.019039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d19afa-2e37-42e1-82dc-5bddf293e0be admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.019316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d19afa-2e37-42e1-82dc-5bddf293e0be admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.019947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d19afa-2e37-42e1-82dc-5bddf293e0be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.021876 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d19afa-2e37-42e1-82dc-5bddf293e0be admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.022771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.023402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.023784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.024061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.024709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.025738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 22:01:56.025738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 22:01:56.025738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 22:01:56.026946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.135219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 634/1241] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/projects => generated 393 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.136255 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d19afa-2e37-42e1-82dc-5bddf293e0be admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.150721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 608/1242] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/projects => generated 383 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.150721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.177310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:56.183244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:56.200476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d19afa-2e37-42e1-82dc-5bddf293e0be admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.205214 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f11390-f740-4c75-b1b3-6367f648be74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.205832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f11390-f740-4c75-b1b3-6367f648be74 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.206114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f11390-f740-4c75-b1b3-6367f648be74 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.206400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f11390-f740-4c75-b1b3-6367f648be74 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.207084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f11390-f740-4c75-b1b3-6367f648be74 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.209187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f11390-f740-4c75-b1b3-6367f648be74 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.209545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7ebb4-abbf-4e37-a443-a75a20322feb admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.214332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.214929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.215238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.215511 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.216217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:01:56.216543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.218122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.219508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 635/1243] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:56 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:56.220719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f11390-f740-4c75-b1b3-6367f648be74 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.230942 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f11390-f740-4c75-b1b3-6367f648be74 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.233855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 609/1244] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:01:56.234710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.242431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1f93a1-e870-4900-9984-f3841d6a1bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.243064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1f93a1-e870-4900-9984-f3841d6a1bec admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.243343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1f93a1-e870-4900-9984-f3841d6a1bec admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.243608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1f93a1-e870-4900-9984-f3841d6a1bec admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.244240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1f93a1-e870-4900-9984-f3841d6a1bec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.244563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b0c617-1377-4168-b07b-616be2818f12 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.246117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1f93a1-e870-4900-9984-f3841d6a1bec admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.249698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.250328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.255364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.255364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.255364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:01:56.255364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.255364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.265941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 610/1245] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:01:56.266824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.277092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159a5a16-b7ca-4f14-adf6-09df98490d16 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.282120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.285119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.285119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.285119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.285119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:01:56.285119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.286552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.297990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 611/1246] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:01:56.298753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.308266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c7838e-2ec0-4e2e-8e90-73717d21dc4a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.312891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f92945-03ba-4516-9555-f9902e1c855c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.313458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f92945-03ba-4516-9555-f9902e1c855c admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.313745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f92945-03ba-4516-9555-f9902e1c855c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.314021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f92945-03ba-4516-9555-f9902e1c855c admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.314639 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f92945-03ba-4516-9555-f9902e1c855c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.316530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f92945-03ba-4516-9555-f9902e1c855c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.342541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 636/1247] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/projects => generated 373 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.342541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1f93a1-e870-4900-9984-f3841d6a1bec admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.398578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 612/1248] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/projects => generated 371 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.401069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1f93a1-e870-4900-9984-f3841d6a1bec admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.402670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f92945-03ba-4516-9555-f9902e1c855c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.460802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8a484b-0cea-4bde-9203-178cb4b99680 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.461392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8a484b-0cea-4bde-9203-178cb4b99680 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.461679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8a484b-0cea-4bde-9203-178cb4b99680 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.461950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8a484b-0cea-4bde-9203-178cb4b99680 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.462594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8a484b-0cea-4bde-9203-178cb4b99680 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.464542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8a484b-0cea-4bde-9203-178cb4b99680 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.465014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f92945-03ba-4516-9555-f9902e1c855c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.469557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d70e5-8b89-47c2-b363-306810be9916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.470152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d70e5-8b89-47c2-b363-306810be9916 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.470452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d70e5-8b89-47c2-b363-306810be9916 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.470724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687d70e5-8b89-47c2-b363-306810be9916 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.471377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687d70e5-8b89-47c2-b363-306810be9916 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.473300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687d70e5-8b89-47c2-b363-306810be9916 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.574396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 637/1249] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/users => generated 297 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.575289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8a484b-0cea-4bde-9203-178cb4b99680 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.585172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8a484b-0cea-4bde-9203-178cb4b99680 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.589806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a9b2e-38b2-42f8-b184-536dc828a2bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.590428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a9b2e-38b2-42f8-b184-536dc828a2bf admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.590707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a9b2e-38b2-42f8-b184-536dc828a2bf admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.591010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6a9b2e-38b2-42f8-b184-536dc828a2bf admin admin] PATH_INFO: `/v3/projects/88309b84283a4052aa1d957047390cb8/users/f9b4be7642f3441ab83cf81e1029e017/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.603671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 613/1250] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/users => generated 292 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.608216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d70e5-8b89-47c2-b363-306810be9916 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.608332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6a9b2e-38b2-42f8-b184-536dc828a2bf admin admin] RBAC: Authorizing `identity:create_grant(project_id=88309b84283a4052aa1d957047390cb8, user_id=f9b4be7642f3441ab83cf81e1029e017, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.612965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad6a9b2e-38b2-42f8-b184-536dc828a2bf admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.619930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687d70e5-8b89-47c2-b363-306810be9916 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.624629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2de2bd-d6eb-4559-8083-d4ec49f562a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.625244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2de2bd-d6eb-4559-8083-d4ec49f562a5 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.625518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2de2bd-d6eb-4559-8083-d4ec49f562a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.625783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2de2bd-d6eb-4559-8083-d4ec49f562a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.626574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2de2bd-d6eb-4559-8083-d4ec49f562a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.628521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2de2bd-d6eb-4559-8083-d4ec49f562a5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.699966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 638/1251] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:56 2022] PUT /identity/v3/projects/88309b84283a4052aa1d957047390cb8/users/f9b4be7642f3441ab83cf81e1029e017/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:56.700794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a9b2e-38b2-42f8-b184-536dc828a2bf admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.724983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:56.734915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 614/1252] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/projects => generated 367 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.735760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2de2bd-d6eb-4559-8083-d4ec49f562a5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.762781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6a9b2e-38b2-42f8-b184-536dc828a2bf admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.785754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:56.801586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2de2bd-d6eb-4559-8083-d4ec49f562a5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.806358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e30339-06dd-4187-84de-22c8d3b500d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.806976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e30339-06dd-4187-84de-22c8d3b500d1 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.807266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e30339-06dd-4187-84de-22c8d3b500d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.807537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e30339-06dd-4187-84de-22c8d3b500d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.808169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e30339-06dd-4187-84de-22c8d3b500d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.810264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e30339-06dd-4187-84de-22c8d3b500d1 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.820241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2f63da-bed2-4277-9ad8-0ddd05879212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:56.820870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f63da-bed2-4277-9ad8-0ddd05879212 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:56.821151 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f63da-bed2-4277-9ad8-0ddd05879212 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:56.821426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2f63da-bed2-4277-9ad8-0ddd05879212 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:56.822054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2f63da-bed2-4277-9ad8-0ddd05879212 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:56.823978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2f63da-bed2-4277-9ad8-0ddd05879212 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:56.904729 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 615/1253] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/projects => generated 377 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.905478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e30339-06dd-4187-84de-22c8d3b500d1 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.934895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 639/1254] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/projects => generated 383 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:56.936255 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2f63da-bed2-4277-9ad8-0ddd05879212 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:56.969322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e30339-06dd-4187-84de-22c8d3b500d1 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:56.998734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2f63da-bed2-4277-9ad8-0ddd05879212 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.004123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d839f714-a100-43c9-b109-28bbeaf80cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.005353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839f714-a100-43c9-b109-28bbeaf80cf5 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.005636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839f714-a100-43c9-b109-28bbeaf80cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.005914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d839f714-a100-43c9-b109-28bbeaf80cf5 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.006544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d839f714-a100-43c9-b109-28bbeaf80cf5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.008497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d839f714-a100-43c9-b109-28bbeaf80cf5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.025224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf53530-7bee-4ab4-bfe3-875fbff39761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.025772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf53530-7bee-4ab4-bfe3-875fbff39761 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.026065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf53530-7bee-4ab4-bfe3-875fbff39761 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.028007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf53530-7bee-4ab4-bfe3-875fbff39761 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.028007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf53530-7bee-4ab4-bfe3-875fbff39761 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.028868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf53530-7bee-4ab4-bfe3-875fbff39761 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.109037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 640/1255] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/users => generated 286 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:57.109864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d839f714-a100-43c9-b109-28bbeaf80cf5 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.120159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d839f714-a100-43c9-b109-28bbeaf80cf5 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.124053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 616/1256] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:56 2022] POST /identity/v3/users => generated 287 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:57.124053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf53530-7bee-4ab4-bfe3-875fbff39761 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.127526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0084a753-1637-42b2-84fd-e867d89501bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.128119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0084a753-1637-42b2-84fd-e867d89501bc admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.128400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0084a753-1637-42b2-84fd-e867d89501bc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.128696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0084a753-1637-42b2-84fd-e867d89501bc admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.129367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0084a753-1637-42b2-84fd-e867d89501bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.131646 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0084a753-1637-42b2-84fd-e867d89501bc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.132374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf53530-7bee-4ab4-bfe3-875fbff39761 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.137032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909ecce0-ffc0-4711-a0be-09b360fb3e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.139701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909ecce0-ffc0-4711-a0be-09b360fb3e96 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.139701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909ecce0-ffc0-4711-a0be-09b360fb3e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.139701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909ecce0-ffc0-4711-a0be-09b360fb3e96 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.139701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909ecce0-ffc0-4711-a0be-09b360fb3e96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.140390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-909ecce0-ffc0-4711-a0be-09b360fb3e96 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.143450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 641/1257] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.144541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c856635-ad10-4d77-a041-6665072a948b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.144820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c856635-ad10-4d77-a041-6665072a948b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.145089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c856635-ad10-4d77-a041-6665072a948b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.150771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 617/1258] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.151647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909ecce0-ffc0-4711-a0be-09b360fb3e96 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.162005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909ecce0-ffc0-4711-a0be-09b360fb3e96 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.164300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3c856635-ad10-4d77-a041-6665072a948b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:57.165521 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3c856635-ad10-4d77-a041-6665072a948b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:57.166815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c0fe1f-04ea-4e3c-81a8-b4516319f104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.167444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0fe1f-04ea-4e3c-81a8-b4516319f104 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.167720 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0fe1f-04ea-4e3c-81a8-b4516319f104 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.167986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c0fe1f-04ea-4e3c-81a8-b4516319f104 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.168603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c0fe1f-04ea-4e3c-81a8-b4516319f104 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.170561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c0fe1f-04ea-4e3c-81a8-b4516319f104 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.173326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 642/1259] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:57.174070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c856635-ad10-4d77-a041-6665072a948b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.184604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c856635-ad10-4d77-a041-6665072a948b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.189343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f0e0af-e9af-44c6-b7f0-0164d41e2758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.191405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0e0af-e9af-44c6-b7f0-0164d41e2758 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.191405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0e0af-e9af-44c6-b7f0-0164d41e2758 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.191405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f0e0af-e9af-44c6-b7f0-0164d41e2758 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.191405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f0e0af-e9af-44c6-b7f0-0164d41e2758 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.193801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f0e0af-e9af-44c6-b7f0-0164d41e2758 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.237403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 618/1260] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:57 2022] POST /identity/v3/users => generated 284 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:57.238237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c0fe1f-04ea-4e3c-81a8-b4516319f104 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.248374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c0fe1f-04ea-4e3c-81a8-b4516319f104 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.253296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44386dd-4d3c-42ed-83aa-a4afbd3bb919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.253915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44386dd-4d3c-42ed-83aa-a4afbd3bb919 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.254197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44386dd-4d3c-42ed-83aa-a4afbd3bb919 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.254469 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c44386dd-4d3c-42ed-83aa-a4afbd3bb919 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.255772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44386dd-4d3c-42ed-83aa-a4afbd3bb919 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.257689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c44386dd-4d3c-42ed-83aa-a4afbd3bb919 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.259609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 643/1261] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:57 2022] POST /identity/v3/users => generated 289 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:57.259609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f0e0af-e9af-44c6-b7f0-0164d41e2758 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.268950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f0e0af-e9af-44c6-b7f0-0164d41e2758 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.273801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b24b9d-7921-4b6b-a13a-c5ff886a73fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.274399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b24b9d-7921-4b6b-a13a-c5ff886a73fd admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.274684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b24b9d-7921-4b6b-a13a-c5ff886a73fd admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.274980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b24b9d-7921-4b6b-a13a-c5ff886a73fd admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.275629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b24b9d-7921-4b6b-a13a-c5ff886a73fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.277246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b24b9d-7921-4b6b-a13a-c5ff886a73fd admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.279517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:57.288755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 644/1262] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.290196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b24b9d-7921-4b6b-a13a-c5ff886a73fd admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.299926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b24b9d-7921-4b6b-a13a-c5ff886a73fd admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.305469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39152617-e0d9-4ff7-831f-ff48d53e845c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.306030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39152617-e0d9-4ff7-831f-ff48d53e845c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.306309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39152617-e0d9-4ff7-831f-ff48d53e845c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.306578 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39152617-e0d9-4ff7-831f-ff48d53e845c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.307227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39152617-e0d9-4ff7-831f-ff48d53e845c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.308755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39152617-e0d9-4ff7-831f-ff48d53e845c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.318357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 645/1263] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.319263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39152617-e0d9-4ff7-831f-ff48d53e845c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.328322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39152617-e0d9-4ff7-831f-ff48d53e845c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.332890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ca8006-a849-418a-9591-87dab86a0ada admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.333445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca8006-a849-418a-9591-87dab86a0ada admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.333749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca8006-a849-418a-9591-87dab86a0ada admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.333991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca8006-a849-418a-9591-87dab86a0ada admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.334605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ca8006-a849-418a-9591-87dab86a0ada admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.336183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5ca8006-a849-418a-9591-87dab86a0ada admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.339646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 619/1264] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:57 2022] POST /identity/v3/users => generated 292 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:57.339646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44386dd-4d3c-42ed-83aa-a4afbd3bb919 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.347091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 646/1265] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.349358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c44386dd-4d3c-42ed-83aa-a4afbd3bb919 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.351586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ca8006-a849-418a-9591-87dab86a0ada admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.354438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dc71c3-e5eb-4674-9d3d-3717043a5eb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.355187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc71c3-e5eb-4674-9d3d-3717043a5eb3 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.355481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc71c3-e5eb-4674-9d3d-3717043a5eb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.355762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dc71c3-e5eb-4674-9d3d-3717043a5eb3 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.356429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dc71c3-e5eb-4674-9d3d-3717043a5eb3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.358099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58dc71c3-e5eb-4674-9d3d-3717043a5eb3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.368961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 620/1266] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.374232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dc71c3-e5eb-4674-9d3d-3717043a5eb3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.379367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58dc71c3-e5eb-4674-9d3d-3717043a5eb3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.384059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7932c-0153-468b-bf45-4604820d0c6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.384642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7932c-0153-468b-bf45-4604820d0c6a admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.384950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7932c-0153-468b-bf45-4604820d0c6a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.385199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49d7932c-0153-468b-bf45-4604820d0c6a admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.385897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d7932c-0153-468b-bf45-4604820d0c6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.387597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49d7932c-0153-468b-bf45-4604820d0c6a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.392885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:57.398464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 621/1267] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.399178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7932c-0153-468b-bf45-4604820d0c6a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.408071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ca8006-a849-418a-9591-87dab86a0ada admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.409524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49d7932c-0153-468b-bf45-4604820d0c6a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.412376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad21500a-e87d-46dc-98b0-4b12f6e10369 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.412924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21500a-e87d-46dc-98b0-4b12f6e10369 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.413203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21500a-e87d-46dc-98b0-4b12f6e10369 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.413479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad21500a-e87d-46dc-98b0-4b12f6e10369 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.415309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd33b17-35e0-491c-b56e-2268cc8c9633 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.415913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd33b17-35e0-491c-b56e-2268cc8c9633 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.416190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd33b17-35e0-491c-b56e-2268cc8c9633 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.416458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd33b17-35e0-491c-b56e-2268cc8c9633 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.417089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd33b17-35e0-491c-b56e-2268cc8c9633 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.418677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd33b17-35e0-491c-b56e-2268cc8c9633 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.422973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad21500a-e87d-46dc-98b0-4b12f6e10369 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.424882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad21500a-e87d-46dc-98b0-4b12f6e10369 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.429602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 622/1268] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.432742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd33b17-35e0-491c-b56e-2268cc8c9633 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.435749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 647/1269] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:01:57.436602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad21500a-e87d-46dc-98b0-4b12f6e10369 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.439815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd33b17-35e0-491c-b56e-2268cc8c9633 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.446104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad21500a-e87d-46dc-98b0-4b12f6e10369 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.447281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e85106-4c33-40f6-bd8b-279ba0eab567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.447881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e85106-4c33-40f6-bd8b-279ba0eab567 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.448156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e85106-4c33-40f6-bd8b-279ba0eab567 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.448423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e85106-4c33-40f6-bd8b-279ba0eab567 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.449067 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e85106-4c33-40f6-bd8b-279ba0eab567 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.450661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e85106-4c33-40f6-bd8b-279ba0eab567 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.454883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aa8f30-a756-43fd-a045-ca72fea1acc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.455551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aa8f30-a756-43fd-a045-ca72fea1acc3 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.455829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aa8f30-a756-43fd-a045-ca72fea1acc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.456096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67aa8f30-a756-43fd-a045-ca72fea1acc3 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.456984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67aa8f30-a756-43fd-a045-ca72fea1acc3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.458525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67aa8f30-a756-43fd-a045-ca72fea1acc3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.463697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 623/1270] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.464529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e85106-4c33-40f6-bd8b-279ba0eab567 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.468687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 648/1271] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.469417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aa8f30-a756-43fd-a045-ca72fea1acc3 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.477107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e85106-4c33-40f6-bd8b-279ba0eab567 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.481501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67aa8f30-a756-43fd-a045-ca72fea1acc3 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.482024 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d4681b-4851-4f78-8033-6b396111e898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.482777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d4681b-4851-4f78-8033-6b396111e898 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.483359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d4681b-4851-4f78-8033-6b396111e898 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.483636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d4681b-4851-4f78-8033-6b396111e898 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.484283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d4681b-4851-4f78-8033-6b396111e898 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.485892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d4681b-4851-4f78-8033-6b396111e898 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.486847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8b57f-b531-467d-9a88-b9ca3c270423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.487692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8b57f-b531-467d-9a88-b9ca3c270423 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.488362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8b57f-b531-467d-9a88-b9ca3c270423 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.488643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8b57f-b531-467d-9a88-b9ca3c270423 admin admin] PATH_INFO: `/v3/projects/65cb45d314f743799145a0fc614ce83e/users/2c7d8acfab1041fc8edf36fd42b06993/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.496870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 624/1272] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.497651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d4681b-4851-4f78-8033-6b396111e898 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.507128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d4681b-4851-4f78-8033-6b396111e898 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.508426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd8b57f-b531-467d-9a88-b9ca3c270423 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65cb45d314f743799145a0fc614ce83e, user_id=2c7d8acfab1041fc8edf36fd42b06993, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.511072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd8b57f-b531-467d-9a88-b9ca3c270423 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.511784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccf9e33-8e68-4588-b2ac-d4765faa0950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.512420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccf9e33-8e68-4588-b2ac-d4765faa0950 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.512700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccf9e33-8e68-4588-b2ac-d4765faa0950 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.513326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccf9e33-8e68-4588-b2ac-d4765faa0950 admin admin] PATH_INFO: `/v3/projects/9d7c346beac34a4aae8b3be8e5489f88/users/e3a3c499785348d487fdbe56819aba65/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.528344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccf9e33-8e68-4588-b2ac-d4765faa0950 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d7c346beac34a4aae8b3be8e5489f88, user_id=e3a3c499785348d487fdbe56819aba65, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.531181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccf9e33-8e68-4588-b2ac-d4765faa0950 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.600905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 649/1273] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:57 2022] PUT /identity/v3/projects/65cb45d314f743799145a0fc614ce83e/users/2c7d8acfab1041fc8edf36fd42b06993/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:57.602158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8b57f-b531-467d-9a88-b9ca3c270423 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.624315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 625/1274] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:57 2022] PUT /identity/v3/projects/9d7c346beac34a4aae8b3be8e5489f88/users/e3a3c499785348d487fdbe56819aba65/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:57.625500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccf9e33-8e68-4588-b2ac-d4765faa0950 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.679044 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8b57f-b531-467d-9a88-b9ca3c270423 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.697684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccf9e33-8e68-4588-b2ac-d4765faa0950 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.702718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c7929-91d4-4e31-a60c-be469d896365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.703360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c7929-91d4-4e31-a60c-be469d896365 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.706657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c7929-91d4-4e31-a60c-be469d896365 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.706657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588c7929-91d4-4e31-a60c-be469d896365 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.706657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588c7929-91d4-4e31-a60c-be469d896365 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.706657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-588c7929-91d4-4e31-a60c-be469d896365 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.720259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 626/1275] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.720259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c7929-91d4-4e31-a60c-be469d896365 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.730550 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588c7929-91d4-4e31-a60c-be469d896365 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.734053 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:57.735488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df4451a-4e38-4c4c-a28a-887cec089b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.736286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df4451a-4e38-4c4c-a28a-887cec089b0d admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.736569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df4451a-4e38-4c4c-a28a-887cec089b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.736846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df4451a-4e38-4c4c-a28a-887cec089b0d admin admin] PATH_INFO: `/v3/projects/a98f7848726149baa0ee68ae4b355c07/users/85c28f1561284c38b8b430d480e4eb89/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.744938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ec9b0-aa8c-4502-bdd6-e511c9d9ace0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.745910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ec9b0-aa8c-4502-bdd6-e511c9d9ace0 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.746365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ec9b0-aa8c-4502-bdd6-e511c9d9ace0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.746791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ec9b0-aa8c-4502-bdd6-e511c9d9ace0 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.747732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ec9b0-aa8c-4502-bdd6-e511c9d9ace0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.749495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ec9b0-aa8c-4502-bdd6-e511c9d9ace0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.758237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df4451a-4e38-4c4c-a28a-887cec089b0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a98f7848726149baa0ee68ae4b355c07, user_id=85c28f1561284c38b8b430d480e4eb89, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.758237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df4451a-4e38-4c4c-a28a-887cec089b0d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.762897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 650/1276] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.763817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ec9b0-aa8c-4502-bdd6-e511c9d9ace0 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.774795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ec9b0-aa8c-4502-bdd6-e511c9d9ace0 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.779450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b56abe3-683e-463a-b861-f4fac9406224 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.780102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56abe3-683e-463a-b861-f4fac9406224 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.780389 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56abe3-683e-463a-b861-f4fac9406224 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.780659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56abe3-683e-463a-b861-f4fac9406224 admin admin] PATH_INFO: `/v3/projects/a91947afbaac424687272654e103fd89/users/1ab8e2ac6f0e48e4aa413613238ad5bf/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.796295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b56abe3-683e-463a-b861-f4fac9406224 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a91947afbaac424687272654e103fd89, user_id=1ab8e2ac6f0e48e4aa413613238ad5bf, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.798967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b56abe3-683e-463a-b861-f4fac9406224 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.823771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 627/1277] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:57 2022] PUT /identity/v3/projects/a98f7848726149baa0ee68ae4b355c07/users/85c28f1561284c38b8b430d480e4eb89/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:57.824624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df4451a-4e38-4c4c-a28a-887cec089b0d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:57.887285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df4451a-4e38-4c4c-a28a-887cec089b0d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:57.892003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a229530-40be-43ef-9c51-952b911edce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:57.892568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a229530-40be-43ef-9c51-952b911edce5 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.892844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a229530-40be-43ef-9c51-952b911edce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.893113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a229530-40be-43ef-9c51-952b911edce5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.893743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a229530-40be-43ef-9c51-952b911edce5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:57.895946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a229530-40be-43ef-9c51-952b911edce5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:57.898602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 651/1278] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:57 2022] PUT /identity/v3/projects/a91947afbaac424687272654e103fd89/users/1ab8e2ac6f0e48e4aa413613238ad5bf/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:57.900545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ea8be-974f-4e15-8522-9c31f8264f51 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.900545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ea8be-974f-4e15-8522-9c31f8264f51 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.900545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a1ea8be-974f-4e15-8522-9c31f8264f51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.913354 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 628/1279] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:57.915035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8abeb60-9d38-4313-9bd3-45560094281b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:57.915506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8abeb60-9d38-4313-9bd3-45560094281b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:57.915782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8abeb60-9d38-4313-9bd3-45560094281b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:57.920299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-1a1ea8be-974f-4e15-8522-9c31f8264f51 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:57.921518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1a1ea8be-974f-4e15-8522-9c31f8264f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:57.937959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:57.946408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e8abeb60-9d38-4313-9bd3-45560094281b None None] MFA Rules not processed for user `e3a3c499785348d487fdbe56819aba65`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:57.947679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e8abeb60-9d38-4313-9bd3-45560094281b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:57.990033 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 652/1280] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:57.990871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1ea8be-974f-4e15-8522-9c31f8264f51 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.009693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a1ea8be-974f-4e15-8522-9c31f8264f51 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.014578 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bfb9a2-53a4-446e-8528-9c6bba3dad78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:58.015384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfb9a2-53a4-446e-8528-9c6bba3dad78 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.015672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfb9a2-53a4-446e-8528-9c6bba3dad78 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.015954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bfb9a2-53a4-446e-8528-9c6bba3dad78 admin admin] PATH_INFO: `/v3/projects/95e2ff128466409e98277d6e1c620fca/users/53639656d3a542c0b822ea8084b2e2d5/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.018988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 629/1281] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:58.021307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8abeb60-9d38-4313-9bd3-45560094281b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.039407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8abeb60-9d38-4313-9bd3-45560094281b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.040461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bfb9a2-53a4-446e-8528-9c6bba3dad78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95e2ff128466409e98277d6e1c620fca, user_id=53639656d3a542c0b822ea8084b2e2d5, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:58.049897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bfb9a2-53a4-446e-8528-9c6bba3dad78 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:58.054254 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84852cf-1948-45f2-a20b-252b31ba817e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:58.055257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84852cf-1948-45f2-a20b-252b31ba817e admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.055728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84852cf-1948-45f2-a20b-252b31ba817e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.056443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84852cf-1948-45f2-a20b-252b31ba817e admin admin] PATH_INFO: `/v3/projects/35145e40009848c0998fdd8058ee7bab/users/eed10466347d4080b34d35e5da98e150/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.075763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84852cf-1948-45f2-a20b-252b31ba817e admin admin] RBAC: Authorizing `identity:create_grant(project_id=35145e40009848c0998fdd8058ee7bab, user_id=eed10466347d4080b34d35e5da98e150, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:58.078490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84852cf-1948-45f2-a20b-252b31ba817e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:58.120867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 653/1282] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:57 2022] PUT /identity/v3/projects/95e2ff128466409e98277d6e1c620fca/users/53639656d3a542c0b822ea8084b2e2d5/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:58.122139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2d5064-60b6-44d3-8579-5fb37fc44395 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.122429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2d5064-60b6-44d3-8579-5fb37fc44395 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.122707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2d5064-60b6-44d3-8579-5fb37fc44395 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.160981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 630/1283] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:58 2022] PUT /identity/v3/projects/35145e40009848c0998fdd8058ee7bab/users/eed10466347d4080b34d35e5da98e150/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:58.161098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-bb2d5064-60b6-44d3-8579-5fb37fc44395 None None] MFA Rules not processed for user `85c28f1561284c38b8b430d480e4eb89`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:58.161192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117abbf4-58ec-4663-9ab3-b759f5369b81 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.161192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117abbf4-58ec-4663-9ab3-b759f5369b81 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.161372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-bb2d5064-60b6-44d3-8579-5fb37fc44395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:58.161472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117abbf4-58ec-4663-9ab3-b759f5369b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.189512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-117abbf4-58ec-4663-9ab3-b759f5369b81 None None] MFA Rules not processed for user `1ab8e2ac6f0e48e4aa413613238ad5bf`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:58.191109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-117abbf4-58ec-4663-9ab3-b759f5369b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:58.241371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 654/1284] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:58.243811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2d5064-60b6-44d3-8579-5fb37fc44395 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.275104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 631/1285] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:58.278521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117abbf4-58ec-4663-9ab3-b759f5369b81 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.279953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:58.283398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-117abbf4-58ec-4663-9ab3-b759f5369b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:58.319631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2d5064-60b6-44d3-8579-5fb37fc44395 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.325773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39df2bd-d915-4e07-aece-570897d11cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:58.326636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39df2bd-d915-4e07-aece-570897d11cc4 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.326938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39df2bd-d915-4e07-aece-570897d11cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.327251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39df2bd-d915-4e07-aece-570897d11cc4 admin admin] PATH_INFO: `/v3/projects/0a6de28c59e9470aa38566fdc2713ba0/users/57b6c2a1a4a248158c5abbabe61ad634/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.342533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39df2bd-d915-4e07-aece-570897d11cc4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a6de28c59e9470aa38566fdc2713ba0, user_id=57b6c2a1a4a248158c5abbabe61ad634, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:58.345604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39df2bd-d915-4e07-aece-570897d11cc4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:58.355480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117abbf4-58ec-4663-9ab3-b759f5369b81 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.359209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85da14fc-b2ce-4aa5-9a57-fb8beb23d746 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:58.359815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da14fc-b2ce-4aa5-9a57-fb8beb23d746 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.360096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da14fc-b2ce-4aa5-9a57-fb8beb23d746 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.360369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85da14fc-b2ce-4aa5-9a57-fb8beb23d746 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.369887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85da14fc-b2ce-4aa5-9a57-fb8beb23d746 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:58.371373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85da14fc-b2ce-4aa5-9a57-fb8beb23d746 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:58.380813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 632/1286] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:01:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:01:58.382016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6a6d04-b1ae-40c6-8949-32b86de3ba05 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.382310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6a6d04-b1ae-40c6-8949-32b86de3ba05 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.382585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6a6d04-b1ae-40c6-8949-32b86de3ba05 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.383873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 633/1287] 10.222.0.20 () {54 vars in 961 bytes} [Wed Oct 19 22:01:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:01:58.384961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff03ced-21cb-49d8-831f-56ebfdbc8c2e None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.385242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff03ced-21cb-49d8-831f-56ebfdbc8c2e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.385513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff03ced-21cb-49d8-831f-56ebfdbc8c2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.405543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8ff03ced-21cb-49d8-831f-56ebfdbc8c2e None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:58.406812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8ff03ced-21cb-49d8-831f-56ebfdbc8c2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:58.418526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 634/1288] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:58.418526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5140fe32-183d-4b02-8e1f-6acee08edc16 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.418526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5140fe32-183d-4b02-8e1f-6acee08edc16 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.418526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5140fe32-183d-4b02-8e1f-6acee08edc16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.426973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:58.431520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 655/1289] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:58 2022] PUT /identity/v3/projects/0a6de28c59e9470aa38566fdc2713ba0/users/57b6c2a1a4a248158c5abbabe61ad634/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:58.437173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39df2bd-d915-4e07-aece-570897d11cc4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.450179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5140fe32-183d-4b02-8e1f-6acee08edc16 None None] MFA Rules not processed for user `eed10466347d4080b34d35e5da98e150`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:58.451477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5140fe32-183d-4b02-8e1f-6acee08edc16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:58.505047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39df2bd-d915-4e07-aece-570897d11cc4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.509906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe5f19d-e930-49fb-acd2-a77ecb1a1695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:58.512047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe5f19d-e930-49fb-acd2-a77ecb1a1695 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.512047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe5f19d-e930-49fb-acd2-a77ecb1a1695 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.512047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe5f19d-e930-49fb-acd2-a77ecb1a1695 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.512047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe5f19d-e930-49fb-acd2-a77ecb1a1695 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:58.514046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe5f19d-e930-49fb-acd2-a77ecb1a1695 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:58.527370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 635/1290] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:58.528660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082051f-45a6-4ed7-852f-dced46d19109 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.528948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082051f-45a6-4ed7-852f-dced46d19109 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.529226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c082051f-45a6-4ed7-852f-dced46d19109 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.530852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 636/1291] 10.222.0.20 () {54 vars in 961 bytes} [Wed Oct 19 22:01:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:01:58.531965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ee171-3e17-4565-aaa1-76b2b3d892bf None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.532240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ee171-3e17-4565-aaa1-76b2b3d892bf None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.532509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8ee171-3e17-4565-aaa1-76b2b3d892bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.571093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ed8ee171-3e17-4565-aaa1-76b2b3d892bf None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:58.572320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ed8ee171-3e17-4565-aaa1-76b2b3d892bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:58.616264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 656/1292] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/projects => generated 373 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:58.617032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe5f19d-e930-49fb-acd2-a77ecb1a1695 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.648682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 637/1293] 10.222.0.20 () {58 vars in 1058 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:58.649763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c26dc7b-774e-44f4-894b-439bc1248d82 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.650049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c26dc7b-774e-44f4-894b-439bc1248d82 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.650356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c26dc7b-774e-44f4-894b-439bc1248d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.673534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3c26dc7b-774e-44f4-894b-439bc1248d82 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:58.674815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3c26dc7b-774e-44f4-894b-439bc1248d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:58.690004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe5f19d-e930-49fb-acd2-a77ecb1a1695 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.694252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60879370-1927-44b3-a059-01df30c71826 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:58.700178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60879370-1927-44b3-a059-01df30c71826 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.700178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60879370-1927-44b3-a059-01df30c71826 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.700178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60879370-1927-44b3-a059-01df30c71826 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.743371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 638/1294] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:58.744427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afaf999-47f8-4f98-b754-3e4b974b3e15 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.744713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afaf999-47f8-4f98-b754-3e4b974b3e15 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.744985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0afaf999-47f8-4f98-b754-3e4b974b3e15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.772117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0afaf999-47f8-4f98-b754-3e4b974b3e15 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:58.772877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60879370-1927-44b3-a059-01df30c71826 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:58.773542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0afaf999-47f8-4f98-b754-3e4b974b3e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:58.775076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60879370-1927-44b3-a059-01df30c71826 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:58.782312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:58.790946 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 657/1295] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:01:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:01:58.792407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60879370-1927-44b3-a059-01df30c71826 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.867254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60879370-1927-44b3-a059-01df30c71826 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.872070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0630b76-b05a-4ad7-9843-156ed3f5cb9b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:58.872689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0630b76-b05a-4ad7-9843-156ed3f5cb9b service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.872971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0630b76-b05a-4ad7-9843-156ed3f5cb9b service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.873249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0630b76-b05a-4ad7-9843-156ed3f5cb9b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.892481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 639/1296] 10.222.0.20 () {58 vars in 1058 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:01:58.893887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afaf999-47f8-4f98-b754-3e4b974b3e15 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.908211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0afaf999-47f8-4f98-b754-3e4b974b3e15 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.920010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50938723-6358-4b27-a959-b1bab5df3545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:58.920010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50938723-6358-4b27-a959-b1bab5df3545 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:58.920010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50938723-6358-4b27-a959-b1bab5df3545 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:58.920010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50938723-6358-4b27-a959-b1bab5df3545 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:58.920564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50938723-6358-4b27-a959-b1bab5df3545 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:58.922221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50938723-6358-4b27-a959-b1bab5df3545 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:58.961072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0630b76-b05a-4ad7-9843-156ed3f5cb9b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:58.962908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0630b76-b05a-4ad7-9843-156ed3f5cb9b service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:58.974432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 658/1297] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:01:58 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:01:58.977740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0630b76-b05a-4ad7-9843-156ed3f5cb9b service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:58.992306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0630b76-b05a-4ad7-9843-156ed3f5cb9b service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:58.999106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979ac03d-e38a-4569-9db7-d41fea38126f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:59.000737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ac03d-e38a-4569-9db7-d41fea38126f service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.001034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ac03d-e38a-4569-9db7-d41fea38126f service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.001365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979ac03d-e38a-4569-9db7-d41fea38126f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.014557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:59.071195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979ac03d-e38a-4569-9db7-d41fea38126f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:59.072964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979ac03d-e38a-4569-9db7-d41fea38126f service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:59.087197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 640/1298] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:01:58 2022] POST /identity/v3/users => generated 293 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:01:59.087197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50938723-6358-4b27-a959-b1bab5df3545 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:59.097059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 659/1299] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:01:58 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:01:59.098278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979ac03d-e38a-4569-9db7-d41fea38126f service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:59.107762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50938723-6358-4b27-a959-b1bab5df3545 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:59.114613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e00c3c-26e5-44da-a3a3-e8b4d496ddd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:59.115382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e00c3c-26e5-44da-a3a3-e8b4d496ddd7 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.115662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e00c3c-26e5-44da-a3a3-e8b4d496ddd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.116016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e00c3c-26e5-44da-a3a3-e8b4d496ddd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.121001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979ac03d-e38a-4569-9db7-d41fea38126f service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:59.136801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0872c217-71e6-4177-b8f9-165371cf951e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:59.137624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0872c217-71e6-4177-b8f9-165371cf951e service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.144284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e00c3c-26e5-44da-a3a3-e8b4d496ddd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:59.144442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0872c217-71e6-4177-b8f9-165371cf951e service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.144442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0872c217-71e6-4177-b8f9-165371cf951e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.144671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e00c3c-26e5-44da-a3a3-e8b4d496ddd7 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:59.162768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 641/1300] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:01:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:01:59.163651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e00c3c-26e5-44da-a3a3-e8b4d496ddd7 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:59.197089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e00c3c-26e5-44da-a3a3-e8b4d496ddd7 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:59.208934 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79766d42-5d75-4ba8-97ba-4f514dc7d565 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:59.209701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79766d42-5d75-4ba8-97ba-4f514dc7d565 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.209839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79766d42-5d75-4ba8-97ba-4f514dc7d565 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.210122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79766d42-5d75-4ba8-97ba-4f514dc7d565 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.210782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79766d42-5d75-4ba8-97ba-4f514dc7d565 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:59.215137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79766d42-5d75-4ba8-97ba-4f514dc7d565 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:59.233662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:59.240455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 642/1301] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:59 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:59.245049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79766d42-5d75-4ba8-97ba-4f514dc7d565 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:59.284402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79766d42-5d75-4ba8-97ba-4f514dc7d565 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:59.300189 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0872c217-71e6-4177-b8f9-165371cf951e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:59.303587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22d2554-aa7f-4a4d-8e0f-0b3ef8de3f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:59.304280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22d2554-aa7f-4a4d-8e0f-0b3ef8de3f11 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.304563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22d2554-aa7f-4a4d-8e0f-0b3ef8de3f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.304842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22d2554-aa7f-4a4d-8e0f-0b3ef8de3f11 admin admin] PATH_INFO: `/v3/projects/57f778213f364d7fabe455c2396bbe3c/users/16cdbf6f43ca4cb5bba079844ff40168/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.306631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0872c217-71e6-4177-b8f9-165371cf951e service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:59.326581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 660/1302] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:01:59 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:01:59.331756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22d2554-aa7f-4a4d-8e0f-0b3ef8de3f11 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57f778213f364d7fabe455c2396bbe3c, user_id=16cdbf6f43ca4cb5bba079844ff40168, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:59.335106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22d2554-aa7f-4a4d-8e0f-0b3ef8de3f11 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:59.469804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94899f05-d83c-4c22-94ba-e4f3b45550f4 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.469991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94899f05-d83c-4c22-94ba-e4f3b45550f4 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.470268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94899f05-d83c-4c22-94ba-e4f3b45550f4 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.475251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 661/1303] 10.222.0.20 () {54 vars in 912 bytes} [Wed Oct 19 22:01:59 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:01:59.482538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691b78f-4330-4ff8-9fc3-0ed7b90e5116 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.482943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691b78f-4330-4ff8-9fc3-0ed7b90e5116 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.483337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7691b78f-4330-4ff8-9fc3-0ed7b90e5116 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.485900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 643/1304] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:59 2022] PUT /identity/v3/projects/57f778213f364d7fabe455c2396bbe3c/users/16cdbf6f43ca4cb5bba079844ff40168/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:01:59.490577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22d2554-aa7f-4a4d-8e0f-0b3ef8de3f11 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:59.522618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-7691b78f-4330-4ff8-9fc3-0ed7b90e5116 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:59.527839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7691b78f-4330-4ff8-9fc3-0ed7b90e5116 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:59.584548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22d2554-aa7f-4a4d-8e0f-0b3ef8de3f11 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:59.602136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a1b011-e7ec-45c6-bf14-5f2fbe81a779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:59.606433 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a1b011-e7ec-45c6-bf14-5f2fbe81a779 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.610451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a1b011-e7ec-45c6-bf14-5f2fbe81a779 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.610451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a1b011-e7ec-45c6-bf14-5f2fbe81a779 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.610451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a1b011-e7ec-45c6-bf14-5f2fbe81a779 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:59.620266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a1b011-e7ec-45c6-bf14-5f2fbe81a779 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:59.651597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 644/1305] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:01:59 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:01:59.655899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a1b011-e7ec-45c6-bf14-5f2fbe81a779 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:59.678231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a1b011-e7ec-45c6-bf14-5f2fbe81a779 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:59.696301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699ceb74-ffd7-4d60-91c3-b30f4e9e6c48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:59.697000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699ceb74-ffd7-4d60-91c3-b30f4e9e6c48 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.697281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699ceb74-ffd7-4d60-91c3-b30f4e9e6c48 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.698015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699ceb74-ffd7-4d60-91c3-b30f4e9e6c48 admin admin] PATH_INFO: `/v3/projects/57f778213f364d7fabe455c2396bbe3c/users/16cdbf6f43ca4cb5bba079844ff40168/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.704029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699ceb74-ffd7-4d60-91c3-b30f4e9e6c48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57f778213f364d7fabe455c2396bbe3c, user_id=16cdbf6f43ca4cb5bba079844ff40168, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:01:59.715643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-699ceb74-ffd7-4d60-91c3-b30f4e9e6c48 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:01:59.726500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 662/1306] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:01:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:01:59.741364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7691b78f-4330-4ff8-9fc3-0ed7b90e5116 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:01:59.759987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 645/1307] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:01:59 2022] PUT /identity/v3/projects/57f778213f364d7fabe455c2396bbe3c/users/16cdbf6f43ca4cb5bba079844ff40168/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:01:59.771836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097343d9-2a1a-4885-afb3-df9d6f3f4706 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.772446 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097343d9-2a1a-4885-afb3-df9d6f3f4706 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.772446 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097343d9-2a1a-4885-afb3-df9d6f3f4706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:01:59.851990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-097343d9-2a1a-4885-afb3-df9d6f3f4706 None None] MFA Rules not processed for user `16cdbf6f43ca4cb5bba079844ff40168`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:01:59.853579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-097343d9-2a1a-4885-afb3-df9d6f3f4706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:01:59.876387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:01:59.877388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7691b78f-4330-4ff8-9fc3-0ed7b90e5116 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:01:59.997206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c37c9b-6621-4961-9eac-8f39c5862e25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:01:59.997818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c37c9b-6621-4961-9eac-8f39c5862e25 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:01:59.999006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c37c9b-6621-4961-9eac-8f39c5862e25 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:01:59.999420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c37c9b-6621-4961-9eac-8f39c5862e25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:00.006827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 646/1308] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:01:59 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:00.008238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262956c4-53fc-4f0b-b6c6-bd2bf94761da None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:00.008528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262956c4-53fc-4f0b-b6c6-bd2bf94761da None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:00.008807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-262956c4-53fc-4f0b-b6c6-bd2bf94761da None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:00.014613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 647/1309] 10.222.0.20 () {54 vars in 912 bytes} [Wed Oct 19 22:02:00 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:02:00.021160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed9053b-5462-43f9-b2ed-ef33d06d8263 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:00.021347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed9053b-5462-43f9-b2ed-ef33d06d8263 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:00.022080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed9053b-5462-43f9-b2ed-ef33d06d8263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:00.054731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:00.065376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6ed9053b-5462-43f9-b2ed-ef33d06d8263 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:00.073133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6ed9053b-5462-43f9-b2ed-ef33d06d8263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:00.100706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c37c9b-6621-4961-9eac-8f39c5862e25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:00.106127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c37c9b-6621-4961-9eac-8f39c5862e25 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:00.138691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 663/1310] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:01:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:00.179926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 648/1311] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:02:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:00.319031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c37c9b-6621-4961-9eac-8f39c5862e25 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:00.332448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c37c9b-6621-4961-9eac-8f39c5862e25 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:00.342723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b3b2e-6703-44ab-a45c-9cfb812a3447 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:00.343688 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b3b2e-6703-44ab-a45c-9cfb812a3447 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:00.345132 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b3b2e-6703-44ab-a45c-9cfb812a3447 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:00.345573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b3b2e-6703-44ab-a45c-9cfb812a3447 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:00.396458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14b3b2e-6703-44ab-a45c-9cfb812a3447 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:00.402129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14b3b2e-6703-44ab-a45c-9cfb812a3447 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:00.419624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 664/1312] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:00 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:02:00.610891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed9053b-5462-43f9-b2ed-ef33d06d8263 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:00.761768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed9053b-5462-43f9-b2ed-ef33d06d8263 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:00.761768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d45e116-1a62-4e0d-b619-dfbda9722f07 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:00.761768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45e116-1a62-4e0d-b619-dfbda9722f07 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:00.761768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45e116-1a62-4e0d-b619-dfbda9722f07 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:00.762777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d45e116-1a62-4e0d-b619-dfbda9722f07 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:00.762777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d45e116-1a62-4e0d-b619-dfbda9722f07 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:00.762777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d45e116-1a62-4e0d-b619-dfbda9722f07 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:00.762777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 649/1313] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:02.850936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b3b2e-6703-44ab-a45c-9cfb812a3447 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:02.994403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b3b2e-6703-44ab-a45c-9cfb812a3447 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:03.004370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:03.005231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:03.005615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:03.006002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] PATH_INFO: `/v3/users/85c28f1561284c38b8b430d480e4eb89` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:03.011550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] RBAC: Authorizing `identity:delete_user(user_id=85c28f1561284c38b8b430d480e4eb89)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:03.019212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:03.345399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85c28f1561284c38b8b430d480e4eb89'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:03.349780 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85c28f1561284c38b8b430d480e4eb89'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:03.374293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85c28f1561284c38b8b430d480e4eb89'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:03.393795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85c28f1561284c38b8b430d480e4eb89'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:03.395192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85c28f1561284c38b8b430d480e4eb89'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:03.497074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 665/1314] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:02 2022] DELETE /identity/v3/users/85c28f1561284c38b8b430d480e4eb89 => generated 0 bytes in 651 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:03.501489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce4dc99-1067-49b3-9796-a92bf6932a24 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:03.502070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce4dc99-1067-49b3-9796-a92bf6932a24 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:03.502256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce4dc99-1067-49b3-9796-a92bf6932a24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:03.565838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-fce4dc99-1067-49b3-9796-a92bf6932a24 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:03.571706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fce4dc99-1067-49b3-9796-a92bf6932a24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:03.741756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 650/1315] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:03.815212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:03.816687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:03.937600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:04.000246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d627d9a-cfb8-4595-8026-dbef35355723 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:04.017199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda10303-4e3f-4fcf-a9dc-39edf18e51d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:04.018125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda10303-4e3f-4fcf-a9dc-39edf18e51d3 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:04.018564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda10303-4e3f-4fcf-a9dc-39edf18e51d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:04.019213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda10303-4e3f-4fcf-a9dc-39edf18e51d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:04.042550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda10303-4e3f-4fcf-a9dc-39edf18e51d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:04.042550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda10303-4e3f-4fcf-a9dc-39edf18e51d3 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:04.081316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 666/1316] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:05.823356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce4dc99-1067-49b3-9796-a92bf6932a24 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:05.825499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fce4dc99-1067-49b3-9796-a92bf6932a24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:05.883443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce4dc99-1067-49b3-9796-a92bf6932a24 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:05.903815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:05.905356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:05.910665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:05.912129 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] PATH_INFO: `/v3/projects/a98f7848726149baa0ee68ae4b355c07` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:05.913784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a98f7848726149baa0ee68ae4b355c07)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:05.916897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:05.958268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:06.086266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a98f7848726149baa0ee68ae4b355c07', 'request_id': 'req-e22a9cd8-5ea3-4f7c-a403-002b8041456c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.193799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a98f7848726149baa0ee68ae4b355c07', 'request_id': 'req-e22a9cd8-5ea3-4f7c-a403-002b8041456c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.210618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a98f7848726149baa0ee68ae4b355c07', 'request_id': 'req-e22a9cd8-5ea3-4f7c-a403-002b8041456c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.243712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] The token cache is being invalidate because project a98f7848726149baa0ee68ae4b355c07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:06.243712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.243712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.243712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 651/1317] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:05 2022] DELETE /identity/v3/projects/a98f7848726149baa0ee68ae4b355c07 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:06.244206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda10303-4e3f-4fcf-a9dc-39edf18e51d3 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:06.249973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda10303-4e3f-4fcf-a9dc-39edf18e51d3 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:06.376984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:06.383596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:06.383596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:06.383596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] PATH_INFO: `/v3/users/16cdbf6f43ca4cb5bba079844ff40168` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:06.383596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] RBAC: Authorizing `identity:delete_user(user_id=16cdbf6f43ca4cb5bba079844ff40168)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:06.383596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:06.699128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16cdbf6f43ca4cb5bba079844ff40168'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.699793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16cdbf6f43ca4cb5bba079844ff40168'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.724332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16cdbf6f43ca4cb5bba079844ff40168'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.734065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16cdbf6f43ca4cb5bba079844ff40168'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.736534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-63019dbc-7cd5-4b94-9d24-a629e7d69449 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16cdbf6f43ca4cb5bba079844ff40168'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:06.808227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 667/1318] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:06 2022] DELETE /identity/v3/users/16cdbf6f43ca4cb5bba079844ff40168 => generated 0 bytes in 571 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:07.505558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:07.505558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:07.556099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22a9cd8-5ea3-4f7c-a403-002b8041456c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:07.684308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:07.684961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:07.685274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:07.685542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] PATH_INFO: `/v3/projects/57f778213f364d7fabe455c2396bbe3c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:07.689780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] RBAC: Authorizing `identity:delete_project(project_id=57f778213f364d7fabe455c2396bbe3c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:07.698403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:07.879649 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57f778213f364d7fabe455c2396bbe3c', 'request_id': 'req-cbc6a867-e416-4120-a9ce-3a0e289de24c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:07.962317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57f778213f364d7fabe455c2396bbe3c', 'request_id': 'req-cbc6a867-e416-4120-a9ce-3a0e289de24c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:07.983973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57f778213f364d7fabe455c2396bbe3c', 'request_id': 'req-cbc6a867-e416-4120-a9ce-3a0e289de24c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:08.012610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] The token cache is being invalidate because project 57f778213f364d7fabe455c2396bbe3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:08.013031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:08.013925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cbc6a867-e416-4120-a9ce-3a0e289de24c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:08.015900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 652/1319] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:07 2022] DELETE /identity/v3/projects/57f778213f364d7fabe455c2396bbe3c => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:08.020445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd522ca9-7602-4dc8-abcc-dfd218a80b11 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.020932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd522ca9-7602-4dc8-abcc-dfd218a80b11 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.021030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd522ca9-7602-4dc8-abcc-dfd218a80b11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:08.068623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-cd522ca9-7602-4dc8-abcc-dfd218a80b11 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:08.070173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cd522ca9-7602-4dc8-abcc-dfd218a80b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:08.174629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:08.204933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 668/1320] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:08.214387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a3812d-4eb3-4911-acd2-234a51ed1e5a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.214929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a3812d-4eb3-4911-acd2-234a51ed1e5a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.215507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a3812d-4eb3-4911-acd2-234a51ed1e5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:08.252874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b6a3812d-4eb3-4911-acd2-234a51ed1e5a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:08.254361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b6a3812d-4eb3-4911-acd2-234a51ed1e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:08.269605 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 653/1321] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:08.271472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd522ca9-7602-4dc8-abcc-dfd218a80b11 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:08.284064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd522ca9-7602-4dc8-abcc-dfd218a80b11 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:08.290847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:08.292440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.292935 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.293425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:08.295092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:08.296058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:08.298706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:08.318710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 669/1322] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:08.323202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a3812d-4eb3-4911-acd2-234a51ed1e5a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:08.341483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a3812d-4eb3-4911-acd2-234a51ed1e5a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:08.352629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db878e7f-e206-4a34-a374-da6cbc33331f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:08.353431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db878e7f-e206-4a34-a374-da6cbc33331f admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.353623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db878e7f-e206-4a34-a374-da6cbc33331f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.354009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db878e7f-e206-4a34-a374-da6cbc33331f admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:08.354775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db878e7f-e206-4a34-a374-da6cbc33331f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:08.358238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db878e7f-e206-4a34-a374-da6cbc33331f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:08.460291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 654/1323] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:08 2022] POST /identity/v3/projects => generated 379 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:08.464290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:08.484058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330ae8c7-b044-4db7-9413-5b5459e5d124 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:08.613393 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f056c7-50bd-4c56-8a7f-ed780231b954 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:08.614836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f056c7-50bd-4c56-8a7f-ed780231b954 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.615713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f056c7-50bd-4c56-8a7f-ed780231b954 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.617064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f056c7-50bd-4c56-8a7f-ed780231b954 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:08.618271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f056c7-50bd-4c56-8a7f-ed780231b954 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:08.625295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f056c7-50bd-4c56-8a7f-ed780231b954 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:08.773322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 670/1324] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:08 2022] POST /identity/v3/users => generated 290 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:08.783774 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db878e7f-e206-4a34-a374-da6cbc33331f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:08.800277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db878e7f-e206-4a34-a374-da6cbc33331f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:08.809901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:08.810064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.810837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.811548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:08.813365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:08.815388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:08.819102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:08.834219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 655/1325] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:08.841566 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f056c7-50bd-4c56-8a7f-ed780231b954 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:08.861139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f056c7-50bd-4c56-8a7f-ed780231b954 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:08.879123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175e66c6-85ab-488f-9a45-61e85a7ceb1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:08.882074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175e66c6-85ab-488f-9a45-61e85a7ceb1f admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.882696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175e66c6-85ab-488f-9a45-61e85a7ceb1f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.883072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-175e66c6-85ab-488f-9a45-61e85a7ceb1f admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:08.885517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175e66c6-85ab-488f-9a45-61e85a7ceb1f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:08.889086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-175e66c6-85ab-488f-9a45-61e85a7ceb1f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:08.899724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:08.900913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:08.918431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 671/1326] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:08.922498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175e66c6-85ab-488f-9a45-61e85a7ceb1f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:08.932514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e25709-808a-470d-9105-9279cbf904b2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:08.946729 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:08.948137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.948137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.948137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] PATH_INFO: `/v3/users/1ab8e2ac6f0e48e4aa413613238ad5bf` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:08.950421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ab8e2ac6f0e48e4aa413613238ad5bf)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:08.951686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: WARNING py.warnings [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 22:02:08.951686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: warnings.warn(msg) Oct 19 22:02:08.951686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: Oct 19 22:02:08.957005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:08.963919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-175e66c6-85ab-488f-9a45-61e85a7ceb1f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:08.992074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12637959-32bd-4da5-9582-91bc88874602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:08.993234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12637959-32bd-4da5-9582-91bc88874602 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:08.993680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12637959-32bd-4da5-9582-91bc88874602 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:08.994143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12637959-32bd-4da5-9582-91bc88874602 admin admin] PATH_INFO: `/v3/projects/a2813a58c1e54bda93b81dd56baa8f75/users/d8a12363b7f24ff988fe938d6ab8ebbb/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:09.018092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12637959-32bd-4da5-9582-91bc88874602 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2813a58c1e54bda93b81dd56baa8f75, user_id=d8a12363b7f24ff988fe938d6ab8ebbb, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:09.026565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12637959-32bd-4da5-9582-91bc88874602 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:09.169456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 672/1327] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:08 2022] PUT /identity/v3/projects/a2813a58c1e54bda93b81dd56baa8f75/users/d8a12363b7f24ff988fe938d6ab8ebbb/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:09.172507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e2e655-a87d-4819-8d2a-686696273684 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:09.172985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e2e655-a87d-4819-8d2a-686696273684 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:09.173582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e2e655-a87d-4819-8d2a-686696273684 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:09.225784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-69e2e655-a87d-4819-8d2a-686696273684 None None] MFA Rules not processed for user `d8a12363b7f24ff988fe938d6ab8ebbb`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:09.227794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-69e2e655-a87d-4819-8d2a-686696273684 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:09.268439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ab8e2ac6f0e48e4aa413613238ad5bf'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:09.346124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 673/1328] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:09 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:09.351420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e2e655-a87d-4819-8d2a-686696273684 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:09.362583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ab8e2ac6f0e48e4aa413613238ad5bf'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:09.403244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ab8e2ac6f0e48e4aa413613238ad5bf'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:09.418196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ab8e2ac6f0e48e4aa413613238ad5bf'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:09.419829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70247d80-0943-4d10-b227-44269cc309b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ab8e2ac6f0e48e4aa413613238ad5bf'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:09.435508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 656/1329] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:08 2022] DELETE /identity/v3/users/1ab8e2ac6f0e48e4aa413613238ad5bf => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:09.439925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3a5b16-4ead-404f-b3c2-cfcc60c2d815 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:09.440256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3a5b16-4ead-404f-b3c2-cfcc60c2d815 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:09.440541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3a5b16-4ead-404f-b3c2-cfcc60c2d815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:09.482782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6f3a5b16-4ead-404f-b3c2-cfcc60c2d815 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:09.482782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6f3a5b16-4ead-404f-b3c2-cfcc60c2d815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:09.493539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e2e655-a87d-4819-8d2a-686696273684 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:09.623381 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:09.627013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:09.627013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:09.627013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:09.627013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:09.635398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:09.697643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 657/1330] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:09.756607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3a5b16-4ead-404f-b3c2-cfcc60c2d815 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:09.767472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6f3a5b16-4ead-404f-b3c2-cfcc60c2d815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:09.787975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:09.790439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 674/1331] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:09 2022] POST /identity/v3/projects => generated 381 bytes in 439 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:09.795456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:09.802533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:09.931420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3a5b16-4ead-404f-b3c2-cfcc60c2d815 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:09.996435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff1f6d7-989b-4ed9-a0ab-591cdec4a454 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:10.008416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372981eb-fea3-44ef-90b6-cc9087f1e1f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:10.011034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372981eb-fea3-44ef-90b6-cc9087f1e1f7 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:10.014851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372981eb-fea3-44ef-90b6-cc9087f1e1f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:10.014851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-372981eb-fea3-44ef-90b6-cc9087f1e1f7 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:10.014851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372981eb-fea3-44ef-90b6-cc9087f1e1f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:10.015909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-372981eb-fea3-44ef-90b6-cc9087f1e1f7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:10.073647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd05a7e-d643-483a-b5d5-f25fc40b4ccb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:10.074845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd05a7e-d643-483a-b5d5-f25fc40b4ccb service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:10.075460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd05a7e-d643-483a-b5d5-f25fc40b4ccb service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:10.077979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd05a7e-d643-483a-b5d5-f25fc40b4ccb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:10.106043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 675/1332] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:09 2022] POST /identity/v3/users => generated 297 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:10.106996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372981eb-fea3-44ef-90b6-cc9087f1e1f7 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:10.115647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd05a7e-d643-483a-b5d5-f25fc40b4ccb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:10.119061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd05a7e-d643-483a-b5d5-f25fc40b4ccb service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:10.132341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-372981eb-fea3-44ef-90b6-cc9087f1e1f7 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:10.152133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c004157-c101-4651-9a8b-8c4d19a4a3cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:10.152963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c004157-c101-4651-9a8b-8c4d19a4a3cb admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:10.153420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c004157-c101-4651-9a8b-8c4d19a4a3cb admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:10.153869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c004157-c101-4651-9a8b-8c4d19a4a3cb admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:10.155383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c004157-c101-4651-9a8b-8c4d19a4a3cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:10.159055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c004157-c101-4651-9a8b-8c4d19a4a3cb admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:10.166345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 658/1333] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:10.199168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 676/1334] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:10 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:10.201752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd05a7e-d643-483a-b5d5-f25fc40b4ccb service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:10.239543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd05a7e-d643-483a-b5d5-f25fc40b4ccb service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:10.253949 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20025af7-fee6-4fb2-863f-7cd142183b3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:10.255322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20025af7-fee6-4fb2-863f-7cd142183b3c admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:10.255708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20025af7-fee6-4fb2-863f-7cd142183b3c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:10.256222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20025af7-fee6-4fb2-863f-7cd142183b3c admin admin] PATH_INFO: `/v3/projects/88de5da494004828858393441b1d3c9c/users/150a5962b3904d25b2ca7a6646e97310/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:10.297975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20025af7-fee6-4fb2-863f-7cd142183b3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=88de5da494004828858393441b1d3c9c, user_id=150a5962b3904d25b2ca7a6646e97310, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:10.300899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20025af7-fee6-4fb2-863f-7cd142183b3c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:10.307528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:10.405239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 659/1335] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:10 2022] PUT /identity/v3/projects/88de5da494004828858393441b1d3c9c/users/150a5962b3904d25b2ca7a6646e97310/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:10.422289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c004157-c101-4651-9a8b-8c4d19a4a3cb admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:10.496374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c004157-c101-4651-9a8b-8c4d19a4a3cb admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:10.620332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:10.667412 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-decfd867-64df-4a4e-a8bf-265e6c28efe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:10.669548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decfd867-64df-4a4e-a8bf-265e6c28efe5 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:10.670354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decfd867-64df-4a4e-a8bf-265e6c28efe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:10.671814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-decfd867-64df-4a4e-a8bf-265e6c28efe5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:10.683313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decfd867-64df-4a4e-a8bf-265e6c28efe5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:10.687716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-decfd867-64df-4a4e-a8bf-265e6c28efe5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:10.729431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 677/1336] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:10 2022] GET /identity/v3/roles => generated 1420 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:10.734158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20025af7-fee6-4fb2-863f-7cd142183b3c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:10.756458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20025af7-fee6-4fb2-863f-7cd142183b3c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:10.766312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:10.767179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:10.773807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:10.773807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] PATH_INFO: `/v3/projects/88de5da494004828858393441b1d3c9c/users/150a5962b3904d25b2ca7a6646e97310/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:10.775913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] RBAC: Authorizing `identity:create_grant(project_id=88de5da494004828858393441b1d3c9c, user_id=150a5962b3904d25b2ca7a6646e97310, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:10.778713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:10.831837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 660/1337] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:10 2022] PUT /identity/v3/projects/88de5da494004828858393441b1d3c9c/users/150a5962b3904d25b2ca7a6646e97310/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:10.843262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c7bf5-f1cc-4689-bd5a-6e528c0baa56 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c7bf5-f1cc-4689-bd5a-6e528c0baa56 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:10.843937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9c7bf5-f1cc-4689-bd5a-6e528c0baa56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:10.952714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-8a9c7bf5-f1cc-4689-bd5a-6e528c0baa56 None None] MFA Rules not processed for user `150a5962b3904d25b2ca7a6646e97310`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:10.952714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8a9c7bf5-f1cc-4689-bd5a-6e528c0baa56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:10.987201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 678/1338] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:10 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:11.005200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:11.015084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:11.152274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b10ec-99e2-4629-8448-80f41c5a3acd admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:11.157163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:11.158090 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:11.158525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:11.158997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:11.237815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:11.237815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:11.237815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 661/1339] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:11 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:11.309551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40e6ad-00c8-449b-aaf6-e65691b119ad None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:11.310442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40e6ad-00c8-449b-aaf6-e65691b119ad None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:11.310884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e40e6ad-00c8-449b-aaf6-e65691b119ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:11.361925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-4e40e6ad-00c8-449b-aaf6-e65691b119ad None None] MFA Rules not processed for user `f9b4be7642f3441ab83cf81e1029e017`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:11.363667 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4e40e6ad-00c8-449b-aaf6-e65691b119ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:11.537373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 679/1340] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:11 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:11.545351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:11.547339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:11.779435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63416199-a17b-4240-804e-ae4e1f55aaaa service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:11.804750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a673187-0eee-4326-b4b0-1a39f77974f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:11.805796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a673187-0eee-4326-b4b0-1a39f77974f7 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:11.806832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a673187-0eee-4326-b4b0-1a39f77974f7 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:11.811613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a673187-0eee-4326-b4b0-1a39f77974f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:11.830160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a673187-0eee-4326-b4b0-1a39f77974f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:11.832111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a673187-0eee-4326-b4b0-1a39f77974f7 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:11.866214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 662/1341] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:11 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:12.396802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40e6ad-00c8-449b-aaf6-e65691b119ad None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:12.398908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4e40e6ad-00c8-449b-aaf6-e65691b119ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:12.528300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:12.608648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e40e6ad-00c8-449b-aaf6-e65691b119ad None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:12.628215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:12.642498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:12.642498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:12.642498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] PATH_INFO: `/v3/users/e3a3c499785348d487fdbe56819aba65` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:12.646719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3a3c499785348d487fdbe56819aba65)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:12.657255 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:12.716831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a673187-0eee-4326-b4b0-1a39f77974f7 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:12.734243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a673187-0eee-4326-b4b0-1a39f77974f7 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:12.737484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '150a5962b3904d25b2ca7a6646e97310', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '88de5da494004828858393441b1d3c9c', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:12.738328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:12.738778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:12.739274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:12.740525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:12.741328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:12.743579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:12.790997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 663/1342] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:02:12.794403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:12.823765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c618c134-62fc-484a-afbf-d0b80494daf3 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:12.858938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '150a5962b3904d25b2ca7a6646e97310', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '88de5da494004828858393441b1d3c9c', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:12.860132 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:12.860418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:12.860692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:12.865541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:12.872366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:12.888815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:12.900057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3a3c499785348d487fdbe56819aba65'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:12.901051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3a3c499785348d487fdbe56819aba65'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:12.928166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3a3c499785348d487fdbe56819aba65'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:12.945987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3a3c499785348d487fdbe56819aba65'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:12.955518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3a3c499785348d487fdbe56819aba65'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:13.043133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 664/1343] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:12 2022] POST /identity/v3/projects => generated 355 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:13.044252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:13.055814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:13.064715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 680/1344] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:12 2022] DELETE /identity/v3/users/e3a3c499785348d487fdbe56819aba65 => generated 0 bytes in 667 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:13.071366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:13.071366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:13.470029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7611537f-1cf0-435a-9b75-24c6ba305645 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:13.476863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2315c440-9bbb-4023-9f1d-95f8b6092cde admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:13.476863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a709a41-d723-4268-a8ab-0eb3503d5143 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '150a5962b3904d25b2ca7a6646e97310', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '88de5da494004828858393441b1d3c9c', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:13.476863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a709a41-d723-4268-a8ab-0eb3503d5143 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:13.482154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a709a41-d723-4268-a8ab-0eb3503d5143 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:13.482154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a709a41-d723-4268-a8ab-0eb3503d5143 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:13.482154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a709a41-d723-4268-a8ab-0eb3503d5143 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:13.482154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a709a41-d723-4268-a8ab-0eb3503d5143 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:13.482154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a709a41-d723-4268-a8ab-0eb3503d5143 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:13.487442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 681/1345] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 410 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:13.487442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bec1d9-e5d1-419a-b6a0-41e0538b1299 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:13.487442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bec1d9-e5d1-419a-b6a0-41e0538b1299 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:13.487442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bec1d9-e5d1-419a-b6a0-41e0538b1299 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:13.509160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:13.510689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:13.511390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:13.512588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] PATH_INFO: `/v3/projects/a91947afbaac424687272654e103fd89` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:13.514043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=a91947afbaac424687272654e103fd89)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:13.517692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:13.520755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e6bec1d9-e5d1-419a-b6a0-41e0538b1299 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:13.522830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e6bec1d9-e5d1-419a-b6a0-41e0538b1299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:13.540384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 682/1346] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:13.568693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bec1d9-e5d1-419a-b6a0-41e0538b1299 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:13.588560 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bec1d9-e5d1-419a-b6a0-41e0538b1299 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:13.593864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '150a5962b3904d25b2ca7a6646e97310', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '88de5da494004828858393441b1d3c9c', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:13.594853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:13.595380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:13.595795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] PATH_INFO: `/v3/projects/a1397f88c9cd4b2694be895fc8da340c` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:13.597201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Authorizing `identity:delete_project(project_id=a1397f88c9cd4b2694be895fc8da340c)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:13.598714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: WARNING py.warnings [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 19 22:02:13.598714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: warnings.warn(msg) Oct 19 22:02:13.598714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: Oct 19 22:02:13.600328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:13.672143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a91947afbaac424687272654e103fd89', 'request_id': 'req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:13.769721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a91947afbaac424687272654e103fd89', 'request_id': 'req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:13.786396 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a91947afbaac424687272654e103fd89', 'request_id': 'req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:13.816904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1397f88c9cd4b2694be895fc8da340c', 'request_id': 'req-1dbb39ad-c370-4dc7-8cf3-09741c879e49'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:13.822116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:13.822289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] The token cache is being invalidate because project a91947afbaac424687272654e103fd89 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:13.822445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:13.827888 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:13.841429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 665/1347] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:13 2022] DELETE /identity/v3/projects/a91947afbaac424687272654e103fd89 => generated 0 bytes in 797 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:13.842892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:13.847579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:13.896210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1397f88c9cd4b2694be895fc8da340c', 'request_id': 'req-1dbb39ad-c370-4dc7-8cf3-09741c879e49'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:13.916820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1397f88c9cd4b2694be895fc8da340c', 'request_id': 'req-1dbb39ad-c370-4dc7-8cf3-09741c879e49'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:14.022092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] The token cache is being invalidate because project a1397f88c9cd4b2694be895fc8da340c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:14.022521 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:14.023380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1dbb39ad-c370-4dc7-8cf3-09741c879e49 tempest-VolumeQuotaClassesTest-2005683509 tempest-VolumeQuotaClassesTest-2005683509-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:14.027436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 683/1348] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:13 2022] DELETE /identity/v3/projects/a1397f88c9cd4b2694be895fc8da340c => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:14.038417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd7fad4-55d3-4480-b45c-f85e3d68a1fb admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:14.038829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b8ae89-a10c-4f72-a352-73899185a8d8 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:14.039417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b8ae89-a10c-4f72-a352-73899185a8d8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:14.041621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b8ae89-a10c-4f72-a352-73899185a8d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:14.112772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-11b8ae89-a10c-4f72-a352-73899185a8d8 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:14.119723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-11b8ae89-a10c-4f72-a352-73899185a8d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:14.216003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3850d4-3c95-4358-a9d4-33d3ed0d4d4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:14.216912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3850d4-3c95-4358-a9d4-33d3ed0d4d4b service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:14.217405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3850d4-3c95-4358-a9d4-33d3ed0d4d4b service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:14.218174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3850d4-3c95-4358-a9d4-33d3ed0d4d4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:14.220357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6e3850d4-3c95-4358-a9d4-33d3ed0d4d4b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:14.314616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 684/1349] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:14.317250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb138cd2-2e77-4647-a75e-0622aac9e3e2 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:14.317942 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb138cd2-2e77-4647-a75e-0622aac9e3e2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:14.318411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb138cd2-2e77-4647-a75e-0622aac9e3e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:14.402822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3850d4-3c95-4358-a9d4-33d3ed0d4d4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:14.404649 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3850d4-3c95-4358-a9d4-33d3ed0d4d4b service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:14.421953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-eb138cd2-2e77-4647-a75e-0622aac9e3e2 None None] MFA Rules not processed for user `57b6c2a1a4a248158c5abbabe61ad634`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:14.424421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-eb138cd2-2e77-4647-a75e-0622aac9e3e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:14.431574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 666/1350] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 589 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:14.433298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dceed1-88a1-49df-9ca8-89d30ece2cc8 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:14.433852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dceed1-88a1-49df-9ca8-89d30ece2cc8 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:14.434199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dceed1-88a1-49df-9ca8-89d30ece2cc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:14.487845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c5dceed1-88a1-49df-9ca8-89d30ece2cc8 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:14.489824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c5dceed1-88a1-49df-9ca8-89d30ece2cc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:14.519419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 667/1351] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:14.523475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dceed1-88a1-49df-9ca8-89d30ece2cc8 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:14.543621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:14.556642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5dceed1-88a1-49df-9ca8-89d30ece2cc8 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:14.591174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:14.591772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:14.592048 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:14.594843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:14.599735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:14.600058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:14.610249 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:14.650414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 685/1352] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 334 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:14.662660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb138cd2-2e77-4647-a75e-0622aac9e3e2 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:14.665715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-eb138cd2-2e77-4647-a75e-0622aac9e3e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:14.668856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 668/1353] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:14.672596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:14.726252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04dfaea-a4ec-4535-ad84-06ac78791cf7 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:14.734429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3281d955-1996-4e6a-85b0-4c73c745ebb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:14.735328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3281d955-1996-4e6a-85b0-4c73c745ebb1 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:14.737349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3281d955-1996-4e6a-85b0-4c73c745ebb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:14.737785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3281d955-1996-4e6a-85b0-4c73c745ebb1 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:14.738595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3281d955-1996-4e6a-85b0-4c73c745ebb1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:14.758234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3281d955-1996-4e6a-85b0-4c73c745ebb1 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:14.989959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 669/1354] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:14 2022] POST /identity/v3/projects => generated 373 bytes in 318 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:14.993062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd28ce7e-46aa-46a6-93c3-c76e26f47752 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:14.997570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd28ce7e-46aa-46a6-93c3-c76e26f47752 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:14.997570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd28ce7e-46aa-46a6-93c3-c76e26f47752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:15.068300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-dd28ce7e-46aa-46a6-93c3-c76e26f47752 None None] MFA Rules not processed for user `53639656d3a542c0b822ea8084b2e2d5`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:15.069979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dd28ce7e-46aa-46a6-93c3-c76e26f47752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:15.077888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb138cd2-2e77-4647-a75e-0622aac9e3e2 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:15.206404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 670/1355] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:15.207534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd28ce7e-46aa-46a6-93c3-c76e26f47752 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:15.209174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dd28ce7e-46aa-46a6-93c3-c76e26f47752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:15.236515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b5f23c-c969-43be-889e-321c2c372cdf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:15.239949 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5f23c-c969-43be-889e-321c2c372cdf service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:15.240436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5f23c-c969-43be-889e-321c2c372cdf service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:15.240847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b5f23c-c969-43be-889e-321c2c372cdf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:15.359556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd28ce7e-46aa-46a6-93c3-c76e26f47752 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:15.372494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:15.373021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:15.373307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:15.373585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] PATH_INFO: `/v3/users/d8a12363b7f24ff988fe938d6ab8ebbb` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:15.374930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8a12363b7f24ff988fe938d6ab8ebbb)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:15.377510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:15.432679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b5f23c-c969-43be-889e-321c2c372cdf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:15.434589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b5f23c-c969-43be-889e-321c2c372cdf service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:15.456287 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 686/1356] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:14 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 795 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:15.460169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b5f23c-c969-43be-889e-321c2c372cdf service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:15.485580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b5f23c-c969-43be-889e-321c2c372cdf service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:15.495915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf37e4f-69af-490e-9dea-3cc2ce516691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:15.496588 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf37e4f-69af-490e-9dea-3cc2ce516691 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:15.496869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf37e4f-69af-490e-9dea-3cc2ce516691 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:15.497142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf37e4f-69af-490e-9dea-3cc2ce516691 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:15.500064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf37e4f-69af-490e-9dea-3cc2ce516691 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:15.503742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf37e4f-69af-490e-9dea-3cc2ce516691 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:15.596759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8a12363b7f24ff988fe938d6ab8ebbb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:15.631933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 687/1357] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:15 2022] POST /identity/v3/users => generated 287 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:15.637190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa0355f-b934-48e5-aafc-d5e557d01bd0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:15.637532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa0355f-b934-48e5-aafc-d5e557d01bd0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:15.637805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa0355f-b934-48e5-aafc-d5e557d01bd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:15.689572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8a12363b7f24ff988fe938d6ab8ebbb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:15.706994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6fa0355f-b934-48e5-aafc-d5e557d01bd0 None None] MFA Rules not processed for user `2c7d8acfab1041fc8edf36fd42b06993`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:15.710841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6fa0355f-b934-48e5-aafc-d5e557d01bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:15.721751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8a12363b7f24ff988fe938d6ab8ebbb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:15.771780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8a12363b7f24ff988fe938d6ab8ebbb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:15.774931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4dfd8e31-27c7-4cc3-8e07-5344e8edcd59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8a12363b7f24ff988fe938d6ab8ebbb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:15.830921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 671/1358] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:15 2022] DELETE /identity/v3/users/d8a12363b7f24ff988fe938d6ab8ebbb => generated 0 bytes in 624 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:15.836467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c00844-25f2-457f-84ab-63a54a19c68d None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:15.836851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c00844-25f2-457f-84ab-63a54a19c68d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:15.837203 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c00844-25f2-457f-84ab-63a54a19c68d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:15.865951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 688/1359] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:15 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:15.868067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa0355f-b934-48e5-aafc-d5e557d01bd0 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:15.879168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6fa0355f-b934-48e5-aafc-d5e557d01bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:15.887233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-24c00844-25f2-457f-84ab-63a54a19c68d None None] MFA Rules not processed for user `53639656d3a542c0b822ea8084b2e2d5`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:15.888559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-24c00844-25f2-457f-84ab-63a54a19c68d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:15.910261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa0355f-b934-48e5-aafc-d5e557d01bd0 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:16.005749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2037997f-cb6f-4627-877f-24517dbcf1a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:16.006645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037997f-cb6f-4627-877f-24517dbcf1a0 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.007875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037997f-cb6f-4627-877f-24517dbcf1a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.007875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2037997f-cb6f-4627-877f-24517dbcf1a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.008672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2037997f-cb6f-4627-877f-24517dbcf1a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:16.010706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2037997f-cb6f-4627-877f-24517dbcf1a0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:16.024113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 689/1360] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:15 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:16.025629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c37985-7679-4e06-ae72-ff42eab4b760 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.026021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c37985-7679-4e06-ae72-ff42eab4b760 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.026597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c37985-7679-4e06-ae72-ff42eab4b760 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.045675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 672/1361] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:15 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:16.047033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc6426-0540-4976-9bf5-3bb75e53e956 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.047281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc6426-0540-4976-9bf5-3bb75e53e956 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.047554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc6426-0540-4976-9bf5-3bb75e53e956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.061309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a3c37985-7679-4e06-ae72-ff42eab4b760 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:16.062837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a3c37985-7679-4e06-ae72-ff42eab4b760 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:16.077917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 690/1362] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:16.080929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c37985-7679-4e06-ae72-ff42eab4b760 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:16.093364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f8bc6426-0540-4976-9bf5-3bb75e53e956 None None] MFA Rules not processed for user `2c7d8acfab1041fc8edf36fd42b06993`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:16.094644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f8bc6426-0540-4976-9bf5-3bb75e53e956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:16.105097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c37985-7679-4e06-ae72-ff42eab4b760 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:16.117203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:16.118041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.118422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.118797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.119580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:16.121320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:16.146976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 691/1363] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:16 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:16.150312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:16.155199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:16.231653 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 673/1364] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:16 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:16.232548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bc6426-0540-4976-9bf5-3bb75e53e956 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:16.237294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f8bc6426-0540-4976-9bf5-3bb75e53e956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:16.400313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bc6426-0540-4976-9bf5-3bb75e53e956 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:16.417108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8352155-a63f-42d3-8fc2-3e43d40554f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:16.417858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8352155-a63f-42d3-8fc2-3e43d40554f1 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.418173 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8352155-a63f-42d3-8fc2-3e43d40554f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.418453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8352155-a63f-42d3-8fc2-3e43d40554f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.436028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd91bccf-62c2-4a32-9126-6d2ea391c723 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:16.445542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8352155-a63f-42d3-8fc2-3e43d40554f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:16.447158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8352155-a63f-42d3-8fc2-3e43d40554f1 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:16.451401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:16.453663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.453964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.454601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.459107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:16.464920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:DELETE=3,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:16.467354 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 674/1365] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:16.468282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8352155-a63f-42d3-8fc2-3e43d40554f1 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:16.481165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:16.482260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8352155-a63f-42d3-8fc2-3e43d40554f1 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:16.483343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:16.497144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e11834-2795-45e7-9e41-2deb24256508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:16.497849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e11834-2795-45e7-9e41-2deb24256508 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.498138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e11834-2795-45e7-9e41-2deb24256508 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.498420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e11834-2795-45e7-9e41-2deb24256508 admin admin] PATH_INFO: `/v3/projects/208d603b98d04e729e6e93d5ae794aea/users/7685537bc1a545628845560fe7b40f51/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.512146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 692/1366] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:16 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:16.514358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:16.518208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:16.536619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e11834-2795-45e7-9e41-2deb24256508 admin admin] RBAC: Authorizing `identity:create_grant(project_id=208d603b98d04e729e6e93d5ae794aea, user_id=7685537bc1a545628845560fe7b40f51, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:16.546225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e11834-2795-45e7-9e41-2deb24256508 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:16.566142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e4ea16-5c4f-40d5-bb95-9784f3002ab4 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:16.576270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:16.578358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.578729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.579111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.598924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:16.598924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:16.620071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 693/1367] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:16 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:16.620945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:16.624562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:16.631363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 675/1368] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:16 2022] PUT /identity/v3/projects/208d603b98d04e729e6e93d5ae794aea/users/7685537bc1a545628845560fe7b40f51/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:16.648751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789fb1d2-7d7f-4526-bad1-ddb6fe7ac9eb None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.649284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789fb1d2-7d7f-4526-bad1-ddb6fe7ac9eb None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.650073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789fb1d2-7d7f-4526-bad1-ddb6fe7ac9eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.659433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf603d5-a167-4417-8c4f-81fbd2c9f232 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:16.729247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-789fb1d2-7d7f-4526-bad1-ddb6fe7ac9eb None None] MFA Rules not processed for user `7685537bc1a545628845560fe7b40f51`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:16.733744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-789fb1d2-7d7f-4526-bad1-ddb6fe7ac9eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:16.880659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:16.881426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:16.881585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:16.881849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] PATH_INFO: `/v3/projects/9d7c346beac34a4aae8b3be8e5489f88` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:16.890693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d7c346beac34a4aae8b3be8e5489f88)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:16.897804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:16.906570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 676/1369] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:16 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:02:17.071268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d7c346beac34a4aae8b3be8e5489f88', 'request_id': 'req-bf979f7e-93c1-4eb8-9716-f633b0474d68'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.077869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:17.091727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d7c346beac34a4aae8b3be8e5489f88', 'request_id': 'req-bf979f7e-93c1-4eb8-9716-f633b0474d68'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.104245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d7c346beac34a4aae8b3be8e5489f88', 'request_id': 'req-bf979f7e-93c1-4eb8-9716-f633b0474d68'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.202526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] The token cache is being invalidate because project 9d7c346beac34a4aae8b3be8e5489f88 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:17.203496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.207582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.216951 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c97f53-99ec-4028-b965-db98371a7b1c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:17.217234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c97f53-99ec-4028-b965-db98371a7b1c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:17.217503 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c97f53-99ec-4028-b965-db98371a7b1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:17.223071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 694/1370] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:16 2022] DELETE /identity/v3/projects/9d7c346beac34a4aae8b3be8e5489f88 => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:17.261229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-52c97f53-99ec-4028-b965-db98371a7b1c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:17.262642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-52c97f53-99ec-4028-b965-db98371a7b1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:17.338571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:17.341587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:17.441339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 677/1371] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:17.450079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c48e7b-39a4-478a-9ec0-e46f51e0eb97 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:17.450408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c48e7b-39a4-478a-9ec0-e46f51e0eb97 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:17.450756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c48e7b-39a4-478a-9ec0-e46f51e0eb97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:17.492334 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-94c48e7b-39a4-478a-9ec0-e46f51e0eb97 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:17.504116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-94c48e7b-39a4-478a-9ec0-e46f51e0eb97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:17.531301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 678/1372] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:17.532080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c48e7b-39a4-478a-9ec0-e46f51e0eb97 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:17.556392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf979f7e-93c1-4eb8-9716-f633b0474d68 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:17.557413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c48e7b-39a4-478a-9ec0-e46f51e0eb97 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:17.562316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:17.563877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:17.563877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:17.563877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:17.564406 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:17.564585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:17.564817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:17.566311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:17.566566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:17.566977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:17.567454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] PATH_INFO: `/v3/users/eed10466347d4080b34d35e5da98e150` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:17.570782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=eed10466347d4080b34d35e5da98e150)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:17.578605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:17.597253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 679/1373] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:17.597855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:17.623100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe90e0c2-a3e1-4504-9515-e213e9ff89b7 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:17.645430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a907d73-579f-4e92-990f-3c145c289166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:17.645833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a907d73-579f-4e92-990f-3c145c289166 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:17.646115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a907d73-579f-4e92-990f-3c145c289166 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:17.646389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a907d73-579f-4e92-990f-3c145c289166 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:17.647071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a907d73-579f-4e92-990f-3c145c289166 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:17.649002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a907d73-579f-4e92-990f-3c145c289166 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:17.750265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 680/1374] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:17 2022] POST /identity/v3/projects => generated 357 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:17.752123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97ed27-509d-45ec-bb96-261b8811099d None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:17.752123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97ed27-509d-45ec-bb96-261b8811099d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:17.752285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97ed27-509d-45ec-bb96-261b8811099d None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:17.753617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 681/1375] 10.222.0.20 () {54 vars in 912 bytes} [Wed Oct 19 22:02:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:02:17.759648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0592ae-5000-4e14-b755-9b4cb6b5418d None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:17.759648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0592ae-5000-4e14-b755-9b4cb6b5418d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:17.759861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d0592ae-5000-4e14-b755-9b4cb6b5418d None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:17.765488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 682/1376] 10.222.0.20 () {54 vars in 912 bytes} [Wed Oct 19 22:02:17 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:02:17.766374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0592ae-5000-4e14-b755-9b4cb6b5418d None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:17.768150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3d0592ae-5000-4e14-b755-9b4cb6b5418d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:17.792141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eed10466347d4080b34d35e5da98e150'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.793797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eed10466347d4080b34d35e5da98e150'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.815583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eed10466347d4080b34d35e5da98e150'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.824885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eed10466347d4080b34d35e5da98e150'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.827720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eed10466347d4080b34d35e5da98e150'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:17.841415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:17.913075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 695/1377] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:17 2022] DELETE /identity/v3/users/eed10466347d4080b34d35e5da98e150 => generated 0 bytes in 575 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:17.914720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:17.918106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:17.919552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d0592ae-5000-4e14-b755-9b4cb6b5418d None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:17.996658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:17.998113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:17.998731 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:17.999193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] PATH_INFO: `/v3/projects/a2813a58c1e54bda93b81dd56baa8f75` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.000567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2813a58c1e54bda93b81dd56baa8f75)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:18.004311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:18.088531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac22eba7-d479-4dbc-97b8-16b0710d0e7d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:18.096041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0939f8d0-230f-4763-aedc-5f3d7a10a45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:18.096672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939f8d0-230f-4763-aedc-5f3d7a10a45d admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.096951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939f8d0-230f-4763-aedc-5f3d7a10a45d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.097223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939f8d0-230f-4763-aedc-5f3d7a10a45d admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.098028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0939f8d0-230f-4763-aedc-5f3d7a10a45d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:18.100752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0939f8d0-230f-4763-aedc-5f3d7a10a45d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:18.117950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:18.190670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2813a58c1e54bda93b81dd56baa8f75', 'request_id': 'req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:18.291278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 696/1378] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:17 2022] POST /identity/v3/users => generated 279 bytes in 377 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:18.294713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148c8930-e126-4c8b-8cad-403fc13e130b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.295272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148c8930-e126-4c8b-8cad-403fc13e130b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.295616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148c8930-e126-4c8b-8cad-403fc13e130b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.312779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2813a58c1e54bda93b81dd56baa8f75', 'request_id': 'req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:18.323469 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2813a58c1e54bda93b81dd56baa8f75', 'request_id': 'req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:18.355850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] The token cache is being invalidate because project a2813a58c1e54bda93b81dd56baa8f75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:18.356152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:18.356838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d3c1a6d5-ee4b-4c57-8ca0-1a56d5110872 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:18.364191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 683/1379] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:17 2022] DELETE /identity/v3/projects/a2813a58c1e54bda93b81dd56baa8f75 => generated 0 bytes in 595 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:18.365748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852862a2-0304-4e46-bd90-6115d0038e47 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.366177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852862a2-0304-4e46-bd90-6115d0038e47 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.366591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852862a2-0304-4e46-bd90-6115d0038e47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.394726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-148c8930-e126-4c8b-8cad-403fc13e130b None None] MFA Rules not processed for user `2f705dc9d2e541f1821dda4b5c7b7709`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:18.396830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-148c8930-e126-4c8b-8cad-403fc13e130b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:18.400431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-852862a2-0304-4e46-bd90-6115d0038e47 None None] MFA Rules not processed for user `2f705dc9d2e541f1821dda4b5c7b7709`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:18.408433 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-852862a2-0304-4e46-bd90-6115d0038e47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:18.505649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 697/1380] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:02:18 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:18.513691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2999154a-1079-467b-a98e-1bb55731c19a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.514140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2999154a-1079-467b-a98e-1bb55731c19a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.514470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2999154a-1079-467b-a98e-1bb55731c19a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.547043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 684/1381] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:02:18 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:18.552559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252c82a8-ca04-4bf3-b624-fd3f5e0f73d5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.552857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252c82a8-ca04-4bf3-b624-fd3f5e0f73d5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.552957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2999154a-1079-467b-a98e-1bb55731c19a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:18.553136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252c82a8-ca04-4bf3-b624-fd3f5e0f73d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.554270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2999154a-1079-467b-a98e-1bb55731c19a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:18.602077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-252c82a8-ca04-4bf3-b624-fd3f5e0f73d5 None None] MFA Rules not processed for user `f9b4be7642f3441ab83cf81e1029e017`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:18.609104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-252c82a8-ca04-4bf3-b624-fd3f5e0f73d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:18.668779 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 698/1382] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:18.669722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2999154a-1079-467b-a98e-1bb55731c19a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:18.674120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2999154a-1079-467b-a98e-1bb55731c19a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:18.692881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2999154a-1079-467b-a98e-1bb55731c19a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:18.699345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:18.699962 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.700233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.700503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.701403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:18.703293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:18.720467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 699/1383] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:18 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:18.721641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:18.723870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:18.742165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415057c6-3009-465f-8e5f-f5d579aeaafb admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:18.747997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:18.748861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.748969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.749161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] PATH_INFO: `/v3/users/150a5962b3904d25b2ca7a6646e97310` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.750443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] RBAC: Authorizing `identity:delete_user(user_id=150a5962b3904d25b2ca7a6646e97310)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:18.752387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:18.755189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 685/1384] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:18 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:18.757443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252c82a8-ca04-4bf3-b624-fd3f5e0f73d5 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:18.759298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-252c82a8-ca04-4bf3-b624-fd3f5e0f73d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:18.789533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252c82a8-ca04-4bf3-b624-fd3f5e0f73d5 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:18.794290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:18.795502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.795502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.795648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.804638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:18.806091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:18.826403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 686/1385] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:18.827399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:18.829241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:18.853463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37d9206c-cc68-453f-bf4e-0bc3d427ef64 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:18.858512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c1e49e-3f78-4c67-8621-51bf27ff8edc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:18.865144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c1e49e-3f78-4c67-8621-51bf27ff8edc service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.865576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c1e49e-3f78-4c67-8621-51bf27ff8edc service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.866007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c1e49e-3f78-4c67-8621-51bf27ff8edc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.878515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c1e49e-3f78-4c67-8621-51bf27ff8edc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:18.881013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c1e49e-3f78-4c67-8621-51bf27ff8edc service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:18.893724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 687/1386] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:18.894806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c1e49e-3f78-4c67-8621-51bf27ff8edc service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:18.905281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c1e49e-3f78-4c67-8621-51bf27ff8edc service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:18.911005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be72f2-58a2-4bfe-bae4-0d82fa72c1c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:18.911005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be72f2-58a2-4bfe-bae4-0d82fa72c1c5 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:18.914470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be72f2-58a2-4bfe-bae4-0d82fa72c1c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:18.914999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25be72f2-58a2-4bfe-bae4-0d82fa72c1c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:18.962011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25be72f2-58a2-4bfe-bae4-0d82fa72c1c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:18.967218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25be72f2-58a2-4bfe-bae4-0d82fa72c1c5 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:18.973666 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '150a5962b3904d25b2ca7a6646e97310'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:18.974378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '150a5962b3904d25b2ca7a6646e97310'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:19.005498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '150a5962b3904d25b2ca7a6646e97310'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:19.006498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 688/1387] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:19.007974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be72f2-58a2-4bfe-bae4-0d82fa72c1c5 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:19.021430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25be72f2-58a2-4bfe-bae4-0d82fa72c1c5 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:19.023033 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '150a5962b3904d25b2ca7a6646e97310'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:19.023745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '150a5962b3904d25b2ca7a6646e97310'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:19.033055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:19.033921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:19.034383 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:19.035080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:19.036314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:19.038662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:19.057724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 689/1388] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:19 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:19.058883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:19.061571 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:19.077395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0120e3-c20e-4eb4-ab3b-1f6e199e5943 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:19.089446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 700/1389] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:18 2022] DELETE /identity/v3/users/150a5962b3904d25b2ca7a6646e97310 => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:19.094692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92b79fc-a508-4a80-955a-4f1461097f77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:19.095357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92b79fc-a508-4a80-955a-4f1461097f77 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:19.095638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92b79fc-a508-4a80-955a-4f1461097f77 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:19.095922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92b79fc-a508-4a80-955a-4f1461097f77 admin admin] PATH_INFO: `/v3/projects/02998fb8bdbf497ea6e815fee827bac9/users/d40a61d7590f49a09cfdfc4e4dedacae/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:19.375224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92b79fc-a508-4a80-955a-4f1461097f77 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02998fb8bdbf497ea6e815fee827bac9, user_id=d40a61d7590f49a09cfdfc4e4dedacae, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:19.378980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92b79fc-a508-4a80-955a-4f1461097f77 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:19.492491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:19.494025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:19.509621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 690/1390] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:19 2022] PUT /identity/v3/projects/02998fb8bdbf497ea6e815fee827bac9/users/d40a61d7590f49a09cfdfc4e4dedacae/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:19.511302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c4dd4-2483-4ee5-9967-b10fbe1dda21 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:19.512047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c4dd4-2483-4ee5-9967-b10fbe1dda21 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:19.512341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17c4dd4-2483-4ee5-9967-b10fbe1dda21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:19.568172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f17c4dd4-2483-4ee5-9967-b10fbe1dda21 None None] MFA Rules not processed for user `d40a61d7590f49a09cfdfc4e4dedacae`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:19.578233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f17c4dd4-2483-4ee5-9967-b10fbe1dda21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:19.636666 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebce624-ad74-4bb4-9bad-d21508154142 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:19.641765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:19.731379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 691/1391] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:19 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:19.734913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c4dd4-2483-4ee5-9967-b10fbe1dda21 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:19.735134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f17c4dd4-2483-4ee5-9967-b10fbe1dda21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:19.812495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:19.813271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:19.813790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:19.814134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:19.817544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:19.844585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17c4dd4-2483-4ee5-9967-b10fbe1dda21 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:19.849956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:19.850599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:19.850873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:19.851164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:19.853156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:19.866342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:19.868358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:19.894906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 692/1392] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:19 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:19.896384 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:19.913914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:19.917216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:19.931073 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 701/1393] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:19 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 439 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:20.088504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-040b731f-c6d6-49c8-ad48-5e25fd430c35 service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:20.101859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3d326-afab-4df5-9cfb-64fa0f06a1a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:20.102783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3d326-afab-4df5-9cfb-64fa0f06a1a4 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:20.103278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3d326-afab-4df5-9cfb-64fa0f06a1a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:20.103767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a3d326-afab-4df5-9cfb-64fa0f06a1a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:20.104597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a3d326-afab-4df5-9cfb-64fa0f06a1a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:20.106822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a3d326-afab-4df5-9cfb-64fa0f06a1a4 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:20.163161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:20.200230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:20.201910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:20.225483 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f53799-2b14-4687-9154-09e62c0e1ac1 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:20.240588 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:20.241422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:20.241954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:20.242244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] PATH_INFO: `/v3/projects/88de5da494004828858393441b1d3c9c` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:20.242770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 693/1394] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:19 2022] POST /identity/v3/projects => generated 357 bytes in 347 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:20.244200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88de5da494004828858393441b1d3c9c)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:20.246981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3d326-afab-4df5-9cfb-64fa0f06a1a4 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:20.249263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:20.528042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a3d326-afab-4df5-9cfb-64fa0f06a1a4 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:20.528042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d098d93-7070-4c17-ad45-653674648925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:20.528042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d098d93-7070-4c17-ad45-653674648925 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:20.528042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d098d93-7070-4c17-ad45-653674648925 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:20.530578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d098d93-7070-4c17-ad45-653674648925 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:20.530578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d098d93-7070-4c17-ad45-653674648925 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:20.530578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d098d93-7070-4c17-ad45-653674648925 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:20.641295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88de5da494004828858393441b1d3c9c', 'request_id': 'req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:20.664124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 694/1395] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:20 2022] POST /identity/v3/users => generated 285 bytes in 412 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:20.664210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88de5da494004828858393441b1d3c9c', 'request_id': 'req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:20.668685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7804903-f70e-4289-9cee-ff553df28512 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:20.669035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7804903-f70e-4289-9cee-ff553df28512 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:20.669320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7804903-f70e-4289-9cee-ff553df28512 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:20.670340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88de5da494004828858393441b1d3c9c', 'request_id': 'req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:20.713312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c7804903-f70e-4289-9cee-ff553df28512 None None] MFA Rules not processed for user `2c7d8acfab1041fc8edf36fd42b06993`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:20.714544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c7804903-f70e-4289-9cee-ff553df28512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:20.779134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 695/1396] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:20 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:20.779976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7804903-f70e-4289-9cee-ff553df28512 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:20.781504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c7804903-f70e-4289-9cee-ff553df28512 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:20.797367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] The token cache is being invalidate because project 88de5da494004828858393441b1d3c9c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:20.797367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:20.798005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:20.800811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 702/1397] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:20 2022] DELETE /identity/v3/projects/88de5da494004828858393441b1d3c9c => generated 0 bytes in 601 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:20.802353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:20.805774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:20.854605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:20.947803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cb1e2-71ad-4b66-a12d-a6713d75a251 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:20.965970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:20.966609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:20.966891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:20.967194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:20.968023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:20.970103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7804903-f70e-4289-9cee-ff553df28512 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:20.975448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:20.983592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:20.985894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:20.986543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:20.992000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] PATH_INFO: `/v3/projects/35145e40009848c0998fdd8058ee7bab` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:20.992000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=35145e40009848c0998fdd8058ee7bab)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:20.997941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:20.998629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 703/1398] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:20 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:21.003664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:21.012454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:21.047335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853eaa27-10e3-4a4c-a8e5-adccee755742 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:21.059977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:21.061672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:21.062025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:21.062459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] PATH_INFO: `/v3/users/7685537bc1a545628845560fe7b40f51` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:21.064036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7685537bc1a545628845560fe7b40f51)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:21.067865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:21.199503 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35145e40009848c0998fdd8058ee7bab', 'request_id': 'req-bccee277-d209-47ad-8df1-a2686269ea3e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.322417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35145e40009848c0998fdd8058ee7bab', 'request_id': 'req-bccee277-d209-47ad-8df1-a2686269ea3e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.331950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35145e40009848c0998fdd8058ee7bab', 'request_id': 'req-bccee277-d209-47ad-8df1-a2686269ea3e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.371229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] The token cache is being invalidate because project 35145e40009848c0998fdd8058ee7bab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:21.371737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.375194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-bccee277-d209-47ad-8df1-a2686269ea3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.382243 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7685537bc1a545628845560fe7b40f51'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.383062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7685537bc1a545628845560fe7b40f51'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.384336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 696/1399] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:20 2022] DELETE /identity/v3/projects/35145e40009848c0998fdd8058ee7bab => generated 0 bytes in 601 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:21.388550 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1267b4-97aa-4084-943a-f1f17f8c2753 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:21.389228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1267b4-97aa-4084-943a-f1f17f8c2753 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:21.389513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1267b4-97aa-4084-943a-f1f17f8c2753 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:21.445935 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7685537bc1a545628845560fe7b40f51'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.453761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5b1267b4-97aa-4084-943a-f1f17f8c2753 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:21.478348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5b1267b4-97aa-4084-943a-f1f17f8c2753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:21.483188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7685537bc1a545628845560fe7b40f51'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.483188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-037c5821-be8c-4141-8688-d09969740cb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7685537bc1a545628845560fe7b40f51'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:21.621756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 704/1400] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:20 2022] DELETE /identity/v3/users/7685537bc1a545628845560fe7b40f51 => generated 0 bytes in 620 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:21.621756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1050f1-f8bd-4800-8543-f857a2be3a9a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:21.621756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1050f1-f8bd-4800-8543-f857a2be3a9a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:21.621756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e1050f1-f8bd-4800-8543-f857a2be3a9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:21.627413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 697/1401] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:21.628470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1267b4-97aa-4084-943a-f1f17f8c2753 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:21.631234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5b1267b4-97aa-4084-943a-f1f17f8c2753 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:21.645965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1267b4-97aa-4084-943a-f1f17f8c2753 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:21.702781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-0e1050f1-f8bd-4800-8543-f857a2be3a9a None None] MFA Rules not processed for user `57b6c2a1a4a248158c5abbabe61ad634`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:21.712549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0e1050f1-f8bd-4800-8543-f857a2be3a9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:21.774088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ebd857f-f46d-495f-9c2c-c475276a6d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:21.776097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd857f-f46d-495f-9c2c-c475276a6d04 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:21.776316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd857f-f46d-495f-9c2c-c475276a6d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:21.776604 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ebd857f-f46d-495f-9c2c-c475276a6d04 admin admin] PATH_INFO: `/v3/projects/265a0e121e7342f7a801a8bf0bf002ba/users/2f489f2d8fd54dd6870c05b4d4e9b387/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:21.811223 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebd857f-f46d-495f-9c2c-c475276a6d04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=265a0e121e7342f7a801a8bf0bf002ba, user_id=2f489f2d8fd54dd6870c05b4d4e9b387, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:21.811223 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ebd857f-f46d-495f-9c2c-c475276a6d04 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:21.824014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 705/1402] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:21.825690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd5327c-a75e-4ff7-92be-be96258f495b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:21.826128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd5327c-a75e-4ff7-92be-be96258f495b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:21.826551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd5327c-a75e-4ff7-92be-be96258f495b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:21.871390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-bbd5327c-a75e-4ff7-92be-be96258f495b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:21.876342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-bbd5327c-a75e-4ff7-92be-be96258f495b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:21.902713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 706/1403] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:21.902713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ae339-692c-4d24-99ec-1a5a94f9c479 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:21.902713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ae339-692c-4d24-99ec-1a5a94f9c479 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:21.902713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28ae339-692c-4d24-99ec-1a5a94f9c479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:21.903046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 698/1404] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:21 2022] PUT /identity/v3/projects/265a0e121e7342f7a801a8bf0bf002ba/users/2f489f2d8fd54dd6870c05b4d4e9b387/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:21.903046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be00da9-1268-41b8-80ef-f031d450d808 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:21.903410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be00da9-1268-41b8-80ef-f031d450d808 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:21.903808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be00da9-1268-41b8-80ef-f031d450d808 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:21.924085 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a28ae339-692c-4d24-99ec-1a5a94f9c479 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:21.925599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a28ae339-692c-4d24-99ec-1a5a94f9c479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:21.954202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1be00da9-1268-41b8-80ef-f031d450d808 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:21.955520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1be00da9-1268-41b8-80ef-f031d450d808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:22.113301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 707/1405] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:22.114939 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0a3f50-d041-417e-ba2b-850ea6da49bb None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.115343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0a3f50-d041-417e-ba2b-850ea6da49bb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.116963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0a3f50-d041-417e-ba2b-850ea6da49bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.122748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 699/1406] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:22.127406 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be00da9-1268-41b8-80ef-f031d450d808 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:22.161674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be00da9-1268-41b8-80ef-f031d450d808 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:22.163875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-7b0a3f50-d041-417e-ba2b-850ea6da49bb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:22.166497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7b0a3f50-d041-417e-ba2b-850ea6da49bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:22.173535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e452e865-2a01-4587-bbad-751e6b2ac151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:22.174279 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e452e865-2a01-4587-bbad-751e6b2ac151 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.174691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e452e865-2a01-4587-bbad-751e6b2ac151 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.174982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e452e865-2a01-4587-bbad-751e6b2ac151 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.175748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e452e865-2a01-4587-bbad-751e6b2ac151 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:22.177963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e452e865-2a01-4587-bbad-751e6b2ac151 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:22.208311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 708/1407] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:22.211773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0a3f50-d041-417e-ba2b-850ea6da49bb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:22.215176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7b0a3f50-d041-417e-ba2b-850ea6da49bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:22.226921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 700/1408] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:22 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:22.227862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e452e865-2a01-4587-bbad-751e6b2ac151 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:22.252424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e452e865-2a01-4587-bbad-751e6b2ac151 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:22.278457 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:22.279705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.280124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.280549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.281841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:22.282295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:22.284579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:22.308904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 701/1409] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:22.311631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:22.325107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6480a806-ec72-48c9-86a1-f566db00bd38 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:22.332597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:22.333297 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.333576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.333839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.334497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:22.335101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:22.341544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:22.364928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 702/1410] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:22.366001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:22.382348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0816001-b60a-4cfe-80d5-d51e9a6c6d70 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:22.391241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dad000-1060-4fd2-b5ca-2874c7d9ea1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:22.391981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dad000-1060-4fd2-b5ca-2874c7d9ea1b admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.392759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dad000-1060-4fd2-b5ca-2874c7d9ea1b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.393112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dad000-1060-4fd2-b5ca-2874c7d9ea1b admin admin] PATH_INFO: `/v3/projects/265a0e121e7342f7a801a8bf0bf002ba/users/2f489f2d8fd54dd6870c05b4d4e9b387/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.396440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dad000-1060-4fd2-b5ca-2874c7d9ea1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=265a0e121e7342f7a801a8bf0bf002ba, user_id=2f489f2d8fd54dd6870c05b4d4e9b387, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:22.400278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dad000-1060-4fd2-b5ca-2874c7d9ea1b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:22.413972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0a3f50-d041-417e-ba2b-850ea6da49bb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:22.419056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 703/1411] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:22 2022] PUT /identity/v3/projects/265a0e121e7342f7a801a8bf0bf002ba/users/2f489f2d8fd54dd6870c05b4d4e9b387/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:22.420332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dad000-1060-4fd2-b5ca-2874c7d9ea1b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:22.515864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:22.521338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361e03c3-9e99-4adf-83cc-133b7717754a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:22.522183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e03c3-9e99-4adf-83cc-133b7717754a service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.522637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e03c3-9e99-4adf-83cc-133b7717754a service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.523116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361e03c3-9e99-4adf-83cc-133b7717754a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.543000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dad000-1060-4fd2-b5ca-2874c7d9ea1b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:22.563438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776332ac-5fa8-4606-beff-488e6c9d44c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:22.564844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776332ac-5fa8-4606-beff-488e6c9d44c2 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.565586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776332ac-5fa8-4606-beff-488e6c9d44c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.572414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776332ac-5fa8-4606-beff-488e6c9d44c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.581355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776332ac-5fa8-4606-beff-488e6c9d44c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:22.585856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776332ac-5fa8-4606-beff-488e6c9d44c2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:22.677790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361e03c3-9e99-4adf-83cc-133b7717754a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:22.681682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361e03c3-9e99-4adf-83cc-133b7717754a service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:22.703371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 709/1412] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 494 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:22.709270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361e03c3-9e99-4adf-83cc-133b7717754a service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:22.729292 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361e03c3-9e99-4adf-83cc-133b7717754a service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:22.753071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886b8aef-cc05-49e8-8de6-aad85f20d151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:22.753703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886b8aef-cc05-49e8-8de6-aad85f20d151 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.753983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886b8aef-cc05-49e8-8de6-aad85f20d151 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.754257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886b8aef-cc05-49e8-8de6-aad85f20d151 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.755481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886b8aef-cc05-49e8-8de6-aad85f20d151 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:22.777683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 704/1413] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:22 2022] POST /identity/v3/projects => generated 381 bytes in 358 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:22.779946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6d882-c8d0-4319-ba35-715ea2cdc3d9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:22.780537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6d882-c8d0-4319-ba35-715ea2cdc3d9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:22.781087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a6d882-c8d0-4319-ba35-715ea2cdc3d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:22.790440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886b8aef-cc05-49e8-8de6-aad85f20d151 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:22.910506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-86a6d882-c8d0-4319-ba35-715ea2cdc3d9 None None] MFA Rules not processed for user `2f489f2d8fd54dd6870c05b4d4e9b387`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:22.911959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-86a6d882-c8d0-4319-ba35-715ea2cdc3d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:22.939004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 710/1414] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:22 2022] POST /identity/v3/projects => generated 403 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:22.940026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886b8aef-cc05-49e8-8de6-aad85f20d151 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.068439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 705/1415] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:22 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 289 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:23.069363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a6d882-c8d0-4319-ba35-715ea2cdc3d9 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.104216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886b8aef-cc05-49e8-8de6-aad85f20d151 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.116247 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.125218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.125218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.125218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.134145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.138096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.215111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a6d882-c8d0-4319-ba35-715ea2cdc3d9 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.227919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea8758e-a7e9-42f6-8c82-b061d366bedf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.230675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea8758e-a7e9-42f6-8c82-b061d366bedf admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.230675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea8758e-a7e9-42f6-8c82-b061d366bedf admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.230675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea8758e-a7e9-42f6-8c82-b061d366bedf admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.231177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea8758e-a7e9-42f6-8c82-b061d366bedf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.233130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea8758e-a7e9-42f6-8c82-b061d366bedf admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.325945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 711/1416] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:22 2022] POST /identity/v3/users => generated 291 bytes in 385 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:23.326986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.332206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:23.429876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 706/1417] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:23 2022] POST /identity/v3/users => generated 302 bytes in 361 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:23.430859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea8758e-a7e9-42f6-8c82-b061d366bedf admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.462757 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea8758e-a7e9-42f6-8c82-b061d366bedf admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.481055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5951723-97e7-491e-bd28-78d0cd6ed8b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.486196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5951723-97e7-491e-bd28-78d0cd6ed8b9 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.486673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5951723-97e7-491e-bd28-78d0cd6ed8b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.491426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5951723-97e7-491e-bd28-78d0cd6ed8b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.492156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5951723-97e7-491e-bd28-78d0cd6ed8b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.494036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5951723-97e7-491e-bd28-78d0cd6ed8b9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.509158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf62d854-e432-4652-b457-1641d761104b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.516040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ed5e65-4094-4e19-bb50-a1741e7e1a0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.517135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed5e65-4094-4e19-bb50-a1741e7e1a0d service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.517612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed5e65-4094-4e19-bb50-a1741e7e1a0d service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.518065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ed5e65-4094-4e19-bb50-a1741e7e1a0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.543064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 707/1418] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:23.544290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5951723-97e7-491e-bd28-78d0cd6ed8b9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.575537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5951723-97e7-491e-bd28-78d0cd6ed8b9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.588992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcdfc91-052e-4c41-98c6-19bc46d1f260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.589969 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcdfc91-052e-4c41-98c6-19bc46d1f260 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.590559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcdfc91-052e-4c41-98c6-19bc46d1f260 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.591211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcdfc91-052e-4c41-98c6-19bc46d1f260 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.598447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcdfc91-052e-4c41-98c6-19bc46d1f260 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.598447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcdfc91-052e-4c41-98c6-19bc46d1f260 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.617591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 708/1419] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:23.618593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcdfc91-052e-4c41-98c6-19bc46d1f260 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.622315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:23.635324 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcdfc91-052e-4c41-98c6-19bc46d1f260 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.661141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a8bf4c-4619-4f66-b41b-8b8d364f47cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.661786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a8bf4c-4619-4f66-b41b-8b8d364f47cd admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.662055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a8bf4c-4619-4f66-b41b-8b8d364f47cd admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.662327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42a8bf4c-4619-4f66-b41b-8b8d364f47cd admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.667879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a8bf4c-4619-4f66-b41b-8b8d364f47cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.674317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42a8bf4c-4619-4f66-b41b-8b8d364f47cd admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.681129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ed5e65-4094-4e19-bb50-a1741e7e1a0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.682681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ed5e65-4094-4e19-bb50-a1741e7e1a0d service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.708559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 709/1420] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:23.712278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a8bf4c-4619-4f66-b41b-8b8d364f47cd admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.717869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 712/1421] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:23 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:23.721867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ed5e65-4094-4e19-bb50-a1741e7e1a0d service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.738247 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ed5e65-4094-4e19-bb50-a1741e7e1a0d service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.753814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42a8bf4c-4619-4f66-b41b-8b8d364f47cd admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.754368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.755734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.756034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.756286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] PATH_INFO: `/v3/projects/cf9bdb37629f46cba8603dd3dc3ca52c/users/9205769ca0f443bf8644ba5fca4e98dd/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.759589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8b3fee-d4d0-490e-a727-08bc8567be5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.760211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8b3fee-d4d0-490e-a727-08bc8567be5f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.760491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8b3fee-d4d0-490e-a727-08bc8567be5f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.760759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8b3fee-d4d0-490e-a727-08bc8567be5f admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.761502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8b3fee-d4d0-490e-a727-08bc8567be5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.763336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8b3fee-d4d0-490e-a727-08bc8567be5f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.792147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 710/1422] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:23 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:23.793494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8b3fee-d4d0-490e-a727-08bc8567be5f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.803437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf9bdb37629f46cba8603dd3dc3ca52c, user_id=9205769ca0f443bf8644ba5fca4e98dd, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.815789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.820116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8b3fee-d4d0-490e-a727-08bc8567be5f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:23.827767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177d61c6-b7f2-4d80-9a92-e4fea5dff013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:23.829619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177d61c6-b7f2-4d80-9a92-e4fea5dff013 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:23.829976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177d61c6-b7f2-4d80-9a92-e4fea5dff013 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:23.830334 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177d61c6-b7f2-4d80-9a92-e4fea5dff013 admin admin] PATH_INFO: `/v3/projects/9dac9e5701d54dc3bc4c69462996927c/users/e253f951fb1949408e9e966178cc782d/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:23.909611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177d61c6-b7f2-4d80-9a92-e4fea5dff013 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9dac9e5701d54dc3bc4c69462996927c, user_id=e253f951fb1949408e9e966178cc782d, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:23.913572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177d61c6-b7f2-4d80-9a92-e4fea5dff013 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:23.916699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 713/1423] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:23 2022] PUT /identity/v3/projects/cf9bdb37629f46cba8603dd3dc3ca52c/users/9205769ca0f443bf8644ba5fca4e98dd/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:23.928682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:23.930736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:24.027092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c939e17c-2871-460c-959b-1976a2bcb5f6 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:24.032041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:24.034453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.034453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.034453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] PATH_INFO: `/v3/projects/208d603b98d04e729e6e93d5ae794aea` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.034696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] RBAC: Authorizing `identity:delete_project(project_id=208d603b98d04e729e6e93d5ae794aea)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:24.036711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:24.072056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 711/1424] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:23 2022] PUT /identity/v3/projects/9dac9e5701d54dc3bc4c69462996927c/users/e253f951fb1949408e9e966178cc782d/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:24.074760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d870359-50e7-4ff6-86e8-b8ab21dd010d None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.074760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d870359-50e7-4ff6-86e8-b8ab21dd010d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.074760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d870359-50e7-4ff6-86e8-b8ab21dd010d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.118489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5d870359-50e7-4ff6-86e8-b8ab21dd010d None None] MFA Rules not processed for user `9205769ca0f443bf8644ba5fca4e98dd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:24.120175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5d870359-50e7-4ff6-86e8-b8ab21dd010d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:24.124111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:24.191801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 712/1425] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:24 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:24.195476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d870359-50e7-4ff6-86e8-b8ab21dd010d None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:24.236263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '208d603b98d04e729e6e93d5ae794aea', 'request_id': 'req-101ee411-331f-4df0-b87b-0919bc80880e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:24.244518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '208d603b98d04e729e6e93d5ae794aea', 'request_id': 'req-101ee411-331f-4df0-b87b-0919bc80880e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:24.250997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '208d603b98d04e729e6e93d5ae794aea', 'request_id': 'req-101ee411-331f-4df0-b87b-0919bc80880e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:24.299027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d870359-50e7-4ff6-86e8-b8ab21dd010d None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:24.367655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] The token cache is being invalidate because project 208d603b98d04e729e6e93d5ae794aea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:24.367655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:24.367655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-101ee411-331f-4df0-b87b-0919bc80880e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:24.369531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 714/1426] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:23 2022] DELETE /identity/v3/projects/208d603b98d04e729e6e93d5ae794aea => generated 0 bytes in 452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:24.371327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:24.372175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.372727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.374245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.377982 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:24.384839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba3ecd3-7167-446a-a052-4597a5ca51d0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.392906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba3ecd3-7167-446a-a052-4597a5ca51d0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.392906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba3ecd3-7167-446a-a052-4597a5ca51d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.419709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-4ba3ecd3-7167-446a-a052-4597a5ca51d0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:24.421293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4ba3ecd3-7167-446a-a052-4597a5ca51d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:24.490420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:24.491996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:24.505098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 713/1427] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:24 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:24.506255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:24.508675 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:24.510075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 715/1428] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:24.511829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d89370-0646-4c3f-9331-67e8e0f72d63 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.524703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d89370-0646-4c3f-9331-67e8e0f72d63 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.526641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d89370-0646-4c3f-9331-67e8e0f72d63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.533572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659573c-067e-42ac-b4ff-3f7350c0db66 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:24.540315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69444748-49b5-4da2-9742-f8b82d4d7ebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:24.541451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69444748-49b5-4da2-9742-f8b82d4d7ebe admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.542891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69444748-49b5-4da2-9742-f8b82d4d7ebe admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.543303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69444748-49b5-4da2-9742-f8b82d4d7ebe admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.544916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69444748-49b5-4da2-9742-f8b82d4d7ebe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:24.548828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69444748-49b5-4da2-9742-f8b82d4d7ebe admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:24.569451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-99d89370-0646-4c3f-9331-67e8e0f72d63 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:24.571276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-99d89370-0646-4c3f-9331-67e8e0f72d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:24.591246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 716/1429] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:24.591246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d89370-0646-4c3f-9331-67e8e0f72d63 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:24.594271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-99d89370-0646-4c3f-9331-67e8e0f72d63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:24.690433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d89370-0646-4c3f-9331-67e8e0f72d63 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:24.697296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47a514-e691-4afb-a39f-585439bfbdb6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:24.697296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47a514-e691-4afb-a39f-585439bfbdb6 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.697296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47a514-e691-4afb-a39f-585439bfbdb6 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.697296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47a514-e691-4afb-a39f-585439bfbdb6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.709996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47a514-e691-4afb-a39f-585439bfbdb6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:24.712306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb47a514-e691-4afb-a39f-585439bfbdb6 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:24.733242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 714/1430] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:24 2022] POST /identity/v3/projects => generated 383 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:24.734583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 717/1431] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:24.735001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615edb1-51c4-4cb9-8e8d-1d2578d2e2d9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.735290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615edb1-51c4-4cb9-8e8d-1d2578d2e2d9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.735560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615edb1-51c4-4cb9-8e8d-1d2578d2e2d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.736911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47a514-e691-4afb-a39f-585439bfbdb6 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:24.747438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb47a514-e691-4afb-a39f-585439bfbdb6 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:24.762808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-4615edb1-51c4-4cb9-8e8d-1d2578d2e2d9 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:24.764088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4615edb1-51c4-4cb9-8e8d-1d2578d2e2d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:24.862934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94191e41-1495-4afd-8417-098d0e7a0016 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:24.863963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94191e41-1495-4afd-8417-098d0e7a0016 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.864394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94191e41-1495-4afd-8417-098d0e7a0016 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.864677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94191e41-1495-4afd-8417-098d0e7a0016 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.865319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94191e41-1495-4afd-8417-098d0e7a0016 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:24.865890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 715/1432] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:24.867593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94191e41-1495-4afd-8417-098d0e7a0016 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:24.871606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4615edb1-51c4-4cb9-8e8d-1d2578d2e2d9 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:24.897117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4615edb1-51c4-4cb9-8e8d-1d2578d2e2d9 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:24.902706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:24.903440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.903829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.904164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.904902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:24.905256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:24.906986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:24.924868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 716/1433] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:02:24.927292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:24.937993 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26b1ba-65a2-4e6a-acf9-7ed253a2b7f3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:24.949497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc8624-492d-4a56-a9fd-7e4e8418d0c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:24.950314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc8624-492d-4a56-a9fd-7e4e8418d0c0 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:24.950620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc8624-492d-4a56-a9fd-7e4e8418d0c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:24.950850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcc8624-492d-4a56-a9fd-7e4e8418d0c0 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:24.951544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcc8624-492d-4a56-a9fd-7e4e8418d0c0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:24.954066 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dcc8624-492d-4a56-a9fd-7e4e8418d0c0 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:24.984561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 718/1434] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:24 2022] POST /identity/v3/users => generated 298 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:24.988519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94191e41-1495-4afd-8417-098d0e7a0016 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:25.014872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94191e41-1495-4afd-8417-098d0e7a0016 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:25.047794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bee69d0-8dac-4e42-83ad-304e5200d966 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:25.049946 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee69d0-8dac-4e42-83ad-304e5200d966 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.050240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee69d0-8dac-4e42-83ad-304e5200d966 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.050518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee69d0-8dac-4e42-83ad-304e5200d966 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.054534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bee69d0-8dac-4e42-83ad-304e5200d966 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:25.056300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bee69d0-8dac-4e42-83ad-304e5200d966 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:25.083831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 719/1435] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:24 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:25.086273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bee69d0-8dac-4e42-83ad-304e5200d966 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:25.097752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 717/1436] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:24 2022] POST /identity/v3/projects => generated 381 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:25.099363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc8624-492d-4a56-a9fd-7e4e8418d0c0 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:25.111345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bee69d0-8dac-4e42-83ad-304e5200d966 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:25.261245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dcc8624-492d-4a56-a9fd-7e4e8418d0c0 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:25.271683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25888ec-caa3-4cdc-98af-3ddecd87ddf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:25.272724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25888ec-caa3-4cdc-98af-3ddecd87ddf9 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.273197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25888ec-caa3-4cdc-98af-3ddecd87ddf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.273495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25888ec-caa3-4cdc-98af-3ddecd87ddf9 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.274326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25888ec-caa3-4cdc-98af-3ddecd87ddf9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:25.277238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25888ec-caa3-4cdc-98af-3ddecd87ddf9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:25.278233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fde30f-82e1-42b7-93cc-b571ceadf99d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:25.278991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fde30f-82e1-42b7-93cc-b571ceadf99d admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.279564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fde30f-82e1-42b7-93cc-b571ceadf99d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.279564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1fde30f-82e1-42b7-93cc-b571ceadf99d admin admin] PATH_INFO: `/v3/projects/3e06bf6754a44786bf34bbc757bfdbcc/users/8db6986f3a0942b680280de332e31afb/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.309242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fde30f-82e1-42b7-93cc-b571ceadf99d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e06bf6754a44786bf34bbc757bfdbcc, user_id=8db6986f3a0942b680280de332e31afb, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:25.314401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1fde30f-82e1-42b7-93cc-b571ceadf99d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:25.374239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 718/1437] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:25 2022] POST /identity/v3/users => generated 291 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:25.378464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25888ec-caa3-4cdc-98af-3ddecd87ddf9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:25.401242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25888ec-caa3-4cdc-98af-3ddecd87ddf9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:25.405043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29457664-ede5-414e-af72-20a9589d46f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:25.406163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29457664-ede5-414e-af72-20a9589d46f3 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.408603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29457664-ede5-414e-af72-20a9589d46f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.409071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29457664-ede5-414e-af72-20a9589d46f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.409946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29457664-ede5-414e-af72-20a9589d46f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:25.412107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29457664-ede5-414e-af72-20a9589d46f3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:25.440112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 720/1438] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:25 2022] PUT /identity/v3/projects/3e06bf6754a44786bf34bbc757bfdbcc/users/8db6986f3a0942b680280de332e31afb/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:25.441145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fde30f-82e1-42b7-93cc-b571ceadf99d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:25.479525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 719/1439] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:25 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:25.480258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29457664-ede5-414e-af72-20a9589d46f3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:25.517360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29457664-ede5-414e-af72-20a9589d46f3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:25.617184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1fde30f-82e1-42b7-93cc-b571ceadf99d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:25.631841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e946ffc-3d8e-4e5b-892c-ede2ca2ad0e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:25.632478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e946ffc-3d8e-4e5b-892c-ede2ca2ad0e2 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.632751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e946ffc-3d8e-4e5b-892c-ede2ca2ad0e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.633024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e946ffc-3d8e-4e5b-892c-ede2ca2ad0e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.633703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e946ffc-3d8e-4e5b-892c-ede2ca2ad0e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:25.635342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e946ffc-3d8e-4e5b-892c-ede2ca2ad0e2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:25.661802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fa92fe-4637-4728-a9d7-26051094cf41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:25.662494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa92fe-4637-4728-a9d7-26051094cf41 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.662848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa92fe-4637-4728-a9d7-26051094cf41 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.663220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa92fe-4637-4728-a9d7-26051094cf41 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.663956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fa92fe-4637-4728-a9d7-26051094cf41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:25.665741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fa92fe-4637-4728-a9d7-26051094cf41 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:25.675811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 721/1440] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:25 2022] GET /identity/v3/roles => generated 1420 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:25.680479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e946ffc-3d8e-4e5b-892c-ede2ca2ad0e2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:25.685237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 720/1441] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:25 2022] GET /identity/v3/roles => generated 1420 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:25.686292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fa92fe-4637-4728-a9d7-26051094cf41 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:25.703800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fa92fe-4637-4728-a9d7-26051094cf41 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:25.709025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1af2d4-5bc5-4c12-96f9-15b1955ed4ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:25.711956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1af2d4-5bc5-4c12-96f9-15b1955ed4ef admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.711956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1af2d4-5bc5-4c12-96f9-15b1955ed4ef admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.711956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1af2d4-5bc5-4c12-96f9-15b1955ed4ef admin admin] PATH_INFO: `/v3/projects/b61360ac8eff44f18d6d5425db0a276f/users/d7130b0645074709a7c5c7373b162cab/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.721626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e946ffc-3d8e-4e5b-892c-ede2ca2ad0e2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:25.745408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f62acf1d-2f52-4632-8454-8a1d5af81d28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:25.745408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62acf1d-2f52-4632-8454-8a1d5af81d28 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.745408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62acf1d-2f52-4632-8454-8a1d5af81d28 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.745408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f62acf1d-2f52-4632-8454-8a1d5af81d28 admin admin] PATH_INFO: `/v3/projects/3e06bf6754a44786bf34bbc757bfdbcc/users/8db6986f3a0942b680280de332e31afb/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.750677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62acf1d-2f52-4632-8454-8a1d5af81d28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e06bf6754a44786bf34bbc757bfdbcc, user_id=8db6986f3a0942b680280de332e31afb, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:25.764825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1af2d4-5bc5-4c12-96f9-15b1955ed4ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=b61360ac8eff44f18d6d5425db0a276f, user_id=d7130b0645074709a7c5c7373b162cab, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:25.767269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f62acf1d-2f52-4632-8454-8a1d5af81d28 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:25.769316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af1af2d4-5bc5-4c12-96f9-15b1955ed4ef admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:25.802587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 722/1442] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:25 2022] PUT /identity/v3/projects/3e06bf6754a44786bf34bbc757bfdbcc/users/8db6986f3a0942b680280de332e31afb/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:25.814065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428740c4-a4b6-496d-be6e-7ed1160b1bc1 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.814373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428740c4-a4b6-496d-be6e-7ed1160b1bc1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.814618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428740c4-a4b6-496d-be6e-7ed1160b1bc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:25.875926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-428740c4-a4b6-496d-be6e-7ed1160b1bc1 None None] MFA Rules not processed for user `8db6986f3a0942b680280de332e31afb`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:25.877203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-428740c4-a4b6-496d-be6e-7ed1160b1bc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:25.889943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 721/1443] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:25 2022] PUT /identity/v3/projects/b61360ac8eff44f18d6d5425db0a276f/users/d7130b0645074709a7c5c7373b162cab/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:25.903828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843195e5-49da-4bc4-8e4e-4b075eb09d10 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:25.904131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843195e5-49da-4bc4-8e4e-4b075eb09d10 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:25.908039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843195e5-49da-4bc4-8e4e-4b075eb09d10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:26.003887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-843195e5-49da-4bc4-8e4e-4b075eb09d10 None None] MFA Rules not processed for user `d7130b0645074709a7c5c7373b162cab`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:26.011062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-843195e5-49da-4bc4-8e4e-4b075eb09d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:26.039120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 723/1444] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:25 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:26.041407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480c3459-d06d-41ec-b50f-c4ecc81ae3a3 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:26.041745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480c3459-d06d-41ec-b50f-c4ecc81ae3a3 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:26.042030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480c3459-d06d-41ec-b50f-c4ecc81ae3a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:26.139238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-480c3459-d06d-41ec-b50f-c4ecc81ae3a3 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:26.141856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-480c3459-d06d-41ec-b50f-c4ecc81ae3a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:26.245878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 722/1445] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:25 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:26.250523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843195e5-49da-4bc4-8e4e-4b075eb09d10 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:26.255475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-843195e5-49da-4bc4-8e4e-4b075eb09d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:26.264237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:26.302316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 724/1446] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:02:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 262 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:26.305017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480c3459-d06d-41ec-b50f-c4ecc81ae3a3 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:26.400555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843195e5-49da-4bc4-8e4e-4b075eb09d10 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:26.407309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897c90ad-f42f-40f0-8822-c392e8ab7dfb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:26.408475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897c90ad-f42f-40f0-8822-c392e8ab7dfb service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:26.408943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897c90ad-f42f-40f0-8822-c392e8ab7dfb service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:26.409391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897c90ad-f42f-40f0-8822-c392e8ab7dfb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:26.427152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480c3459-d06d-41ec-b50f-c4ecc81ae3a3 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:26.432518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4197aa0-f337-4b2c-8b3f-8b76eddc344b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:26.433232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4197aa0-f337-4b2c-8b3f-8b76eddc344b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:26.433583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4197aa0-f337-4b2c-8b3f-8b76eddc344b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:26.433938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4197aa0-f337-4b2c-8b3f-8b76eddc344b admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:26.434700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4197aa0-f337-4b2c-8b3f-8b76eddc344b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:26.437185 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4197aa0-f337-4b2c-8b3f-8b76eddc344b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:26.550620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897c90ad-f42f-40f0-8822-c392e8ab7dfb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:26.552963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897c90ad-f42f-40f0-8822-c392e8ab7dfb service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:26.562872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 725/1447] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:26 2022] POST /identity/v3/projects => generated 381 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:26.564098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4197aa0-f337-4b2c-8b3f-8b76eddc344b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:26.565449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 723/1448] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:26 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:26.680120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4197aa0-f337-4b2c-8b3f-8b76eddc344b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:26.686528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:26.687286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:26.687632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:26.687974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:26.689917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:26.693177 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:26.718859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897c90ad-f42f-40f0-8822-c392e8ab7dfb service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:26.730641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897c90ad-f42f-40f0-8822-c392e8ab7dfb service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:26.832573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 726/1449] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:26 2022] POST /identity/v3/users => generated 297 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:26.832573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:26.832573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:26.897376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:26.898304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:26.899100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:26.899380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:26.909817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:27.040961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:27.054099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:27.060585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:27.082444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 724/1450] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:26 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:02:27.084028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:27.104105 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e53a88ac-925d-4c97-b2bd-378dc3559212 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:27.108314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66ec316-4210-416f-a077-16853a0ec8b3 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:27.115305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dc622c-f063-4c53-8e24-1cc50367d886 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:27.115843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dc622c-f063-4c53-8e24-1cc50367d886 service swift] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:27.115945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da977ec-205c-4978-a0c6-9daf8a48560d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:27.116137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dc622c-f063-4c53-8e24-1cc50367d886 service swift] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:27.116388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dc622c-f063-4c53-8e24-1cc50367d886 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:27.116608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da977ec-205c-4978-a0c6-9daf8a48560d admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:27.117099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da977ec-205c-4978-a0c6-9daf8a48560d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:27.117487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da977ec-205c-4978-a0c6-9daf8a48560d admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:27.118276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da977ec-205c-4978-a0c6-9daf8a48560d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:27.120401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da977ec-205c-4978-a0c6-9daf8a48560d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:27.137818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 725/1451] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:27 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:27.140980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6839e50c-e234-4c05-8ea7-17d7346c639e None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:27.142686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6839e50c-e234-4c05-8ea7-17d7346c639e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:27.150207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6839e50c-e234-4c05-8ea7-17d7346c639e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:27.212803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6839e50c-e234-4c05-8ea7-17d7346c639e None None] MFA Rules not processed for user `e253f951fb1949408e9e966178cc782d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:27.218684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6839e50c-e234-4c05-8ea7-17d7346c639e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:27.219670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dc622c-f063-4c53-8e24-1cc50367d886 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:27.222678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dc622c-f063-4c53-8e24-1cc50367d886 service swift] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:27.232811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 727/1452] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:02:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:02:27.233898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dc622c-f063-4c53-8e24-1cc50367d886 service swift] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:27.252591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dc622c-f063-4c53-8e24-1cc50367d886 service swift] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:27.262129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0add83-64fe-40d0-85c1-eb8971a1a521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:27.262910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0add83-64fe-40d0-85c1-eb8971a1a521 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:27.266599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0add83-64fe-40d0-85c1-eb8971a1a521 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:27.267282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0add83-64fe-40d0-85c1-eb8971a1a521 admin admin] PATH_INFO: `/v3/projects/575422ec43664a49a38036f433a91bc8/users/b2238e7f727145d3998b048a367297c0/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:27.312766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0add83-64fe-40d0-85c1-eb8971a1a521 admin admin] RBAC: Authorizing `identity:create_grant(project_id=575422ec43664a49a38036f433a91bc8, user_id=b2238e7f727145d3998b048a367297c0, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:27.350057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0add83-64fe-40d0-85c1-eb8971a1a521 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:27.371736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 726/1453] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:27 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:27.376789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6839e50c-e234-4c05-8ea7-17d7346c639e None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:27.378238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6839e50c-e234-4c05-8ea7-17d7346c639e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:27.399880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6839e50c-e234-4c05-8ea7-17d7346c639e None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:27.405443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c67b02-ee95-4cb6-b4e8-51c0a1ffa955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:27.406015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c67b02-ee95-4cb6-b4e8-51c0a1ffa955 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:27.406345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c67b02-ee95-4cb6-b4e8-51c0a1ffa955 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:27.406619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c67b02-ee95-4cb6-b4e8-51c0a1ffa955 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:27.407378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c67b02-ee95-4cb6-b4e8-51c0a1ffa955 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:27.409375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c67b02-ee95-4cb6-b4e8-51c0a1ffa955 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:27.665943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 728/1454] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:27 2022] PUT /identity/v3/projects/575422ec43664a49a38036f433a91bc8/users/b2238e7f727145d3998b048a367297c0/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:27.669935 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0add83-64fe-40d0-85c1-eb8971a1a521 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:27.671238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 727/1455] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:27 2022] POST /identity/v3/projects => generated 405 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:27.675950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c67b02-ee95-4cb6-b4e8-51c0a1ffa955 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:27.865006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0add83-64fe-40d0-85c1-eb8971a1a521 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:27.935001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c67b02-ee95-4cb6-b4e8-51c0a1ffa955 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:27.945434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8eb3c3-98e1-4638-b3de-0dc6094de44a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:27.946329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8eb3c3-98e1-4638-b3de-0dc6094de44a admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:27.947240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8eb3c3-98e1-4638-b3de-0dc6094de44a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:27.949295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8eb3c3-98e1-4638-b3de-0dc6094de44a admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:27.950116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8eb3c3-98e1-4638-b3de-0dc6094de44a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:27.952864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8eb3c3-98e1-4638-b3de-0dc6094de44a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:27.993408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8d2f56-aab4-4d90-b0b5-1ce85e4162e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:27.994410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8d2f56-aab4-4d90-b0b5-1ce85e4162e3 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:27.994864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8d2f56-aab4-4d90-b0b5-1ce85e4162e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:27.995289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8d2f56-aab4-4d90-b0b5-1ce85e4162e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:27.996401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8d2f56-aab4-4d90-b0b5-1ce85e4162e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:27.998594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8d2f56-aab4-4d90-b0b5-1ce85e4162e3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:28.012549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 729/1456] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:27 2022] GET /identity/v3/roles => generated 1420 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:02:28.017333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8d2f56-aab4-4d90-b0b5-1ce85e4162e3 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:28.052652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8d2f56-aab4-4d90-b0b5-1ce85e4162e3 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:28.061358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4691d9a0-a384-4ee3-9b37-e719a7ed01b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:28.062259 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4691d9a0-a384-4ee3-9b37-e719a7ed01b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:28.062696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4691d9a0-a384-4ee3-9b37-e719a7ed01b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:28.066282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4691d9a0-a384-4ee3-9b37-e719a7ed01b1 admin admin] PATH_INFO: `/v3/projects/575422ec43664a49a38036f433a91bc8/users/b2238e7f727145d3998b048a367297c0/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:28.101027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4691d9a0-a384-4ee3-9b37-e719a7ed01b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=575422ec43664a49a38036f433a91bc8, user_id=b2238e7f727145d3998b048a367297c0, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:28.109695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4691d9a0-a384-4ee3-9b37-e719a7ed01b1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:28.165996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 728/1457] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:27 2022] POST /identity/v3/users => generated 309 bytes in 491 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:28.174180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8eb3c3-98e1-4638-b3de-0dc6094de44a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:28.198824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 730/1458] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:28 2022] PUT /identity/v3/projects/575422ec43664a49a38036f433a91bc8/users/b2238e7f727145d3998b048a367297c0/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:28.202568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722d125c-1803-473c-9ccc-349aee08849a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:28.203805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722d125c-1803-473c-9ccc-349aee08849a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:28.204153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722d125c-1803-473c-9ccc-349aee08849a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:28.222236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8eb3c3-98e1-4638-b3de-0dc6094de44a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:28.288702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-722d125c-1803-473c-9ccc-349aee08849a None None] MFA Rules not processed for user `b2238e7f727145d3998b048a367297c0`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:28.302656 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-722d125c-1803-473c-9ccc-349aee08849a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:28.418013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5137959e-09b5-4c6f-8b35-98d7fcad979b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:28.418013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5137959e-09b5-4c6f-8b35-98d7fcad979b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:28.418013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5137959e-09b5-4c6f-8b35-98d7fcad979b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:28.418013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5137959e-09b5-4c6f-8b35-98d7fcad979b admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:28.418347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5137959e-09b5-4c6f-8b35-98d7fcad979b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:28.418347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5137959e-09b5-4c6f-8b35-98d7fcad979b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:28.450556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 729/1459] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:28.451777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5137959e-09b5-4c6f-8b35-98d7fcad979b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:28.475656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5137959e-09b5-4c6f-8b35-98d7fcad979b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:28.488579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe7e89c-b446-4ced-b5ff-629629a3bf18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:28.490679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e89c-b446-4ced-b5ff-629629a3bf18 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:28.490980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e89c-b446-4ced-b5ff-629629a3bf18 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:28.491268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe7e89c-b446-4ced-b5ff-629629a3bf18 admin admin] PATH_INFO: `/v3/projects/39d16d1c39f64a118a38b1a7060891af/users/f67337ac08f444f1ac48ce642e7938d7/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:28.571668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe7e89c-b446-4ced-b5ff-629629a3bf18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39d16d1c39f64a118a38b1a7060891af, user_id=f67337ac08f444f1ac48ce642e7938d7, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:28.579477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe7e89c-b446-4ced-b5ff-629629a3bf18 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:28.667956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 731/1460] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:28 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 451 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:02:28.667956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650ad38-47be-494a-af7e-00d21b3b5522 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:28.667956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650ad38-47be-494a-af7e-00d21b3b5522 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:28.667956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650ad38-47be-494a-af7e-00d21b3b5522 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:28.737302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6650ad38-47be-494a-af7e-00d21b3b5522 None None] MFA Rules not processed for user `2f705dc9d2e541f1821dda4b5c7b7709`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:28.739537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6650ad38-47be-494a-af7e-00d21b3b5522 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:28.748217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 730/1461] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:28 2022] PUT /identity/v3/projects/39d16d1c39f64a118a38b1a7060891af/users/f67337ac08f444f1ac48ce642e7938d7/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:28.755198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe7e89c-b446-4ced-b5ff-629629a3bf18 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:28.905103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 732/1462] 10.222.0.13 () {58 vars in 1015 bytes} [Wed Oct 19 22:02:28 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:28.911711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6650ad38-47be-494a-af7e-00d21b3b5522 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:28.942780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe7e89c-b446-4ced-b5ff-629629a3bf18 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:28.947891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e93732-d58f-48eb-9995-6656aa2595bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:28.948826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e93732-d58f-48eb-9995-6656aa2595bf service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:28.949942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e93732-d58f-48eb-9995-6656aa2595bf service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:28.950745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e93732-d58f-48eb-9995-6656aa2595bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:28.999013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=4,keystone:SELECT=94,keystone:DELETE=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:29.097317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6650ad38-47be-494a-af7e-00d21b3b5522 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:29.104763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e93732-d58f-48eb-9995-6656aa2595bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:29.107819 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e93732-d58f-48eb-9995-6656aa2595bf service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:29.110037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b298ef-224c-4382-8f69-5c77745c5673 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:29.111609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b298ef-224c-4382-8f69-5c77745c5673 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:29.117864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b298ef-224c-4382-8f69-5c77745c5673 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:29.117864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b298ef-224c-4382-8f69-5c77745c5673 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:29.117864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b298ef-224c-4382-8f69-5c77745c5673 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:29.122431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b298ef-224c-4382-8f69-5c77745c5673 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:29.126874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 731/1463] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:28 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:29.128266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e93732-d58f-48eb-9995-6656aa2595bf service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:29.147209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e93732-d58f-48eb-9995-6656aa2595bf service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:29.165152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c4afca-a37a-4e83-a8b5-5f0b0546247a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:29.173042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:29.187755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 733/1464] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:29.188755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4afca-a37a-4e83-a8b5-5f0b0546247a service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:29.189148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4afca-a37a-4e83-a8b5-5f0b0546247a service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:29.189441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c4afca-a37a-4e83-a8b5-5f0b0546247a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:29.199499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b298ef-224c-4382-8f69-5c77745c5673 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:29.218981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c4afca-a37a-4e83-a8b5-5f0b0546247a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:29.220606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c4afca-a37a-4e83-a8b5-5f0b0546247a service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:29.225817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b298ef-224c-4382-8f69-5c77745c5673 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:29.255915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b278db3c-e048-4d42-86af-c509e894b0f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:29.258302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b278db3c-e048-4d42-86af-c509e894b0f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:29.258736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b278db3c-e048-4d42-86af-c509e894b0f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:29.259354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b278db3c-e048-4d42-86af-c509e894b0f8 admin admin] PATH_INFO: `/v3/projects/39d16d1c39f64a118a38b1a7060891af/users/f67337ac08f444f1ac48ce642e7938d7/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:29.295107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 732/1465] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:29 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:29.295475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b278db3c-e048-4d42-86af-c509e894b0f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=39d16d1c39f64a118a38b1a7060891af, user_id=f67337ac08f444f1ac48ce642e7938d7, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:29.295475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b278db3c-e048-4d42-86af-c509e894b0f8 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:29.372275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 734/1466] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:29 2022] PUT /identity/v3/projects/39d16d1c39f64a118a38b1a7060891af/users/f67337ac08f444f1ac48ce642e7938d7/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:29.431537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c4afca-a37a-4e83-a8b5-5f0b0546247a service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:29.450279 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c4afca-a37a-4e83-a8b5-5f0b0546247a service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:29.686772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67fb9777-badb-4926-960f-bd776894c36c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:29.688463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fb9777-badb-4926-960f-bd776894c36c service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:29.688463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fb9777-badb-4926-960f-bd776894c36c service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:29.688821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fb9777-badb-4926-960f-bd776894c36c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:29.946635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fb9777-badb-4926-960f-bd776894c36c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:29.949566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67fb9777-badb-4926-960f-bd776894c36c service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:29.989605 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 733/1467] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:29 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 559 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:31.303125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:31.557465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=3,keystone:SELECT=96,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:31.792822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b278db3c-e048-4d42-86af-c509e894b0f8 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:31.949506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b278db3c-e048-4d42-86af-c509e894b0f8 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:32.179010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:32.179010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:32.179010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:32.179010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] PATH_INFO: `/v3/users/9205769ca0f443bf8644ba5fca4e98dd` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:32.181402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] RBAC: Authorizing `identity:delete_user(user_id=9205769ca0f443bf8644ba5fca4e98dd)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:32.181402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:32.515579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9205769ca0f443bf8644ba5fca4e98dd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:32.516749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9205769ca0f443bf8644ba5fca4e98dd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:32.551642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9205769ca0f443bf8644ba5fca4e98dd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:32.571448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9205769ca0f443bf8644ba5fca4e98dd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:32.572934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ca5e20b6-3570-465d-b826-b2d956d8280e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9205769ca0f443bf8644ba5fca4e98dd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:32.604566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a28c9-93d9-4777-9f3d-6eae06629f38 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:32.605156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a28c9-93d9-4777-9f3d-6eae06629f38 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:32.605156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244a28c9-93d9-4777-9f3d-6eae06629f38 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:32.606689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 734/1468] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:02:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 22:02:32.640720 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c02d66-834d-40ab-b81c-1047c563f6b6 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:32.640999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c02d66-834d-40ab-b81c-1047c563f6b6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:32.641275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c02d66-834d-40ab-b81c-1047c563f6b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:32.673296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c8c02d66-834d-40ab-b81c-1047c563f6b6 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:32.674560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c8c02d66-834d-40ab-b81c-1047c563f6b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:32.710318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 735/1469] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:31 2022] DELETE /identity/v3/users/9205769ca0f443bf8644ba5fca4e98dd => generated 0 bytes in 913 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:32.715568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2156f54-7f2f-43e3-93e7-eb1390230fd8 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:32.718051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2156f54-7f2f-43e3-93e7-eb1390230fd8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:32.719113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2156f54-7f2f-43e3-93e7-eb1390230fd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:32.753107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c2156f54-7f2f-43e3-93e7-eb1390230fd8 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:32.755132 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c2156f54-7f2f-43e3-93e7-eb1390230fd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:32.839409 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 736/1470] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:32.847370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 735/1471] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:02:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:02:33.068548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2156f54-7f2f-43e3-93e7-eb1390230fd8 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:33.068548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c2156f54-7f2f-43e3-93e7-eb1390230fd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:33.069551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c02d66-834d-40ab-b81c-1047c563f6b6 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:33.069551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c8c02d66-834d-40ab-b81c-1047c563f6b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:33.069551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8c02d66-834d-40ab-b81c-1047c563f6b6 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:33.069764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2156f54-7f2f-43e3-93e7-eb1390230fd8 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:33.069764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36934f0a-d780-4f97-9dff-c844e34507dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:33.069764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36934f0a-d780-4f97-9dff-c844e34507dc service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:33.069764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36934f0a-d780-4f97-9dff-c844e34507dc service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:33.070040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36934f0a-d780-4f97-9dff-c844e34507dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:33.070040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36934f0a-d780-4f97-9dff-c844e34507dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:33.070040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36934f0a-d780-4f97-9dff-c844e34507dc service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:33.116141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 737/1472] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:33.160415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcad089-6037-4a61-a33d-c6cc22f1b1e6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:33.161918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcad089-6037-4a61-a33d-c6cc22f1b1e6 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:33.162200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcad089-6037-4a61-a33d-c6cc22f1b1e6 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:33.162471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcad089-6037-4a61-a33d-c6cc22f1b1e6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:33.180170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcad089-6037-4a61-a33d-c6cc22f1b1e6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:33.182134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bcad089-6037-4a61-a33d-c6cc22f1b1e6 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:33.210332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 736/1473] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:02:33.882615 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:34.070196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d6f951-ca37-413d-a86b-8e3b2b1b2700 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:34.070196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d6f951-ca37-413d-a86b-8e3b2b1b2700 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:34.070196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d6f951-ca37-413d-a86b-8e3b2b1b2700 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:34.070196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 738/1474] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:02:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:02:34.071012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f64cc-50dd-4a4a-950d-a0a1b6fa07d5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:34.071012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f64cc-50dd-4a4a-950d-a0a1b6fa07d5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:34.071012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7f64cc-50dd-4a4a-950d-a0a1b6fa07d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:34.071012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1a7f64cc-50dd-4a4a-950d-a0a1b6fa07d5 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:34.071012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1a7f64cc-50dd-4a4a-950d-a0a1b6fa07d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:34.071012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 737/1475] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:02:33 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:34.102261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d6f951-ca37-413d-a86b-8e3b2b1b2700 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:34.128037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d6f951-ca37-413d-a86b-8e3b2b1b2700 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:34.138247 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:34.139763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:34.140431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:34.141341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:34.158727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:34.162568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:34.184886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 739/1476] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:34 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:34.350023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:35.440477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd26ad-3538-4959-8f9a-5b9796d99949 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:35.445034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd26ad-3538-4959-8f9a-5b9796d99949 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:35.445533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bd26ad-3538-4959-8f9a-5b9796d99949 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:35.528130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a6bd26ad-3538-4959-8f9a-5b9796d99949 None None] MFA Rules not processed for user `f67337ac08f444f1ac48ce642e7938d7`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:35.649506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a6bd26ad-3538-4959-8f9a-5b9796d99949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:35.649506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 738/1477] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:35 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:35.989982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:35.991470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:36.019041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d412eef-0d5f-45b4-8ec6-8c3d38db3a30 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:36.037750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:36.041483 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:36.043253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:36.043838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] PATH_INFO: `/v3/projects/cf9bdb37629f46cba8603dd3dc3ca52c` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:36.047583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf9bdb37629f46cba8603dd3dc3ca52c)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:36.061259 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:36.432947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf9bdb37629f46cba8603dd3dc3ca52c', 'request_id': 'req-3ebae983-98d6-441e-ac95-a7b6577de249'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:36.449095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf9bdb37629f46cba8603dd3dc3ca52c', 'request_id': 'req-3ebae983-98d6-441e-ac95-a7b6577de249'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:36.509175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf9bdb37629f46cba8603dd3dc3ca52c', 'request_id': 'req-3ebae983-98d6-441e-ac95-a7b6577de249'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:36.627332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] The token cache is being invalidate because project cf9bdb37629f46cba8603dd3dc3ca52c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:36.627825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:36.635259 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3ebae983-98d6-441e-ac95-a7b6577de249 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:36.637217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 740/1478] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:35 2022] DELETE /identity/v3/projects/cf9bdb37629f46cba8603dd3dc3ca52c => generated 0 bytes in 648 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:36.641538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bd26ad-3538-4959-8f9a-5b9796d99949 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:36.676330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bd26ad-3538-4959-8f9a-5b9796d99949 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:36.796037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:36.797048 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:36.797541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:36.798045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] PATH_INFO: `/v3/users/8db6986f3a0942b680280de332e31afb` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:36.815087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8db6986f3a0942b680280de332e31afb)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:36.815087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:37.300628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8db6986f3a0942b680280de332e31afb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:37.385144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8db6986f3a0942b680280de332e31afb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:37.399933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8db6986f3a0942b680280de332e31afb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:37.407772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8db6986f3a0942b680280de332e31afb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:37.408781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-563ab874-6b55-4c7f-9670-dc407e94dc5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8db6986f3a0942b680280de332e31afb'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:37.414876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 739/1479] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:36 2022] DELETE /identity/v3/users/8db6986f3a0942b680280de332e31afb => generated 0 bytes in 774 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:37.542727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1acca6d-2bc8-46ef-9e34-57ba4bd6435a None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:37.543256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1acca6d-2bc8-46ef-9e34-57ba4bd6435a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:37.543351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1acca6d-2bc8-46ef-9e34-57ba4bd6435a None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:37.544928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 741/1480] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:02:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:02:37.557130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff6fb35-2068-49e9-bae6-99344dbeb68f None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:37.557497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff6fb35-2068-49e9-bae6-99344dbeb68f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:37.557849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff6fb35-2068-49e9-bae6-99344dbeb68f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:37.580984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-aff6fb35-2068-49e9-bae6-99344dbeb68f None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:37.583039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-aff6fb35-2068-49e9-bae6-99344dbeb68f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:37.655714 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 740/1481] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:02:37 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:37.666826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1acca6d-2bc8-46ef-9e34-57ba4bd6435a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:37.668722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b1acca6d-2bc8-46ef-9e34-57ba4bd6435a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:37.686194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1acca6d-2bc8-46ef-9e34-57ba4bd6435a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:37.691452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec822ff-5207-4195-833b-ba3329f2ba91 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:37.692640 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec822ff-5207-4195-833b-ba3329f2ba91 service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:37.692927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec822ff-5207-4195-833b-ba3329f2ba91 service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:37.693199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec822ff-5207-4195-833b-ba3329f2ba91 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:37.701916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec822ff-5207-4195-833b-ba3329f2ba91 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:37.703552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec822ff-5207-4195-833b-ba3329f2ba91 service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:37.744637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 742/1482] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:37 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:38.318717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff6fb35-2068-49e9-bae6-99344dbeb68f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:38.320344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-aff6fb35-2068-49e9-bae6-99344dbeb68f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:38.415150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff6fb35-2068-49e9-bae6-99344dbeb68f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:38.420245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:38.420959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:38.421327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:38.421763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] PATH_INFO: `/v3/projects/3e06bf6754a44786bf34bbc757bfdbcc` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:38.423275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e06bf6754a44786bf34bbc757bfdbcc)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:38.430098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:38.483491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:38.545879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e06bf6754a44786bf34bbc757bfdbcc', 'request_id': 'req-d8628654-ea13-4f7a-a4d2-21ffa59999cf'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:38.620297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:38.676470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e06bf6754a44786bf34bbc757bfdbcc', 'request_id': 'req-d8628654-ea13-4f7a-a4d2-21ffa59999cf'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:38.683572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e06bf6754a44786bf34bbc757bfdbcc', 'request_id': 'req-d8628654-ea13-4f7a-a4d2-21ffa59999cf'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:38.696378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] The token cache is being invalidate because project 3e06bf6754a44786bf34bbc757bfdbcc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:38.696952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:38.697882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d8628654-ea13-4f7a-a4d2-21ffa59999cf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:38.700153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 741/1483] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:38 2022] DELETE /identity/v3/projects/3e06bf6754a44786bf34bbc757bfdbcc => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:38.705578 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae344ea-1398-417a-b803-5cdb6b4ef21c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:38.705866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae344ea-1398-417a-b803-5cdb6b4ef21c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:38.706140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae344ea-1398-417a-b803-5cdb6b4ef21c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:38.726837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-0ae344ea-1398-417a-b803-5cdb6b4ef21c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:38.728433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0ae344ea-1398-417a-b803-5cdb6b4ef21c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:38.796264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 743/1484] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:38.804035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425cc8d1-6e66-4e17-8e1f-e363277ee05c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:38.804398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425cc8d1-6e66-4e17-8e1f-e363277ee05c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:38.804735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425cc8d1-6e66-4e17-8e1f-e363277ee05c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:38.827839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae344ea-1398-417a-b803-5cdb6b4ef21c None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:38.829262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0ae344ea-1398-417a-b803-5cdb6b4ef21c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:38.829885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-425cc8d1-6e66-4e17-8e1f-e363277ee05c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:38.834608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-425cc8d1-6e66-4e17-8e1f-e363277ee05c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:38.843589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae344ea-1398-417a-b803-5cdb6b4ef21c None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:38.846827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 742/1485] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:38.851063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425cc8d1-6e66-4e17-8e1f-e363277ee05c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:38.863209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:38.863209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:38.863209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:38.863209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] PATH_INFO: `/v3/users/e253f951fb1949408e9e966178cc782d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:38.863586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e253f951fb1949408e9e966178cc782d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:38.863679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425cc8d1-6e66-4e17-8e1f-e363277ee05c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:38.863790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:38.867304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:38.868719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:38.868999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:38.869323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:38.869984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:38.870285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:38.872103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:38.885734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 743/1486] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:02:38.895743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:38.906825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eedfd4-3a77-4663-b261-0d6bda45a006 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:38.915058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc19bc3-80e2-4d9a-938b-d7a3cef293b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:38.915058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc19bc3-80e2-4d9a-938b-d7a3cef293b5 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:38.915058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc19bc3-80e2-4d9a-938b-d7a3cef293b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:38.915058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dc19bc3-80e2-4d9a-938b-d7a3cef293b5 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:38.916030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc19bc3-80e2-4d9a-938b-d7a3cef293b5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:38.916030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dc19bc3-80e2-4d9a-938b-d7a3cef293b5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:39.011800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 744/1487] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:38 2022] POST /identity/v3/projects => generated 385 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:39.012835 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc19bc3-80e2-4d9a-938b-d7a3cef293b5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:39.076126 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dc19bc3-80e2-4d9a-938b-d7a3cef293b5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:39.081095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4709949e-3897-4156-a3ad-2c89ee0d1588 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:39.081713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4709949e-3897-4156-a3ad-2c89ee0d1588 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:39.081987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4709949e-3897-4156-a3ad-2c89ee0d1588 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:39.082283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4709949e-3897-4156-a3ad-2c89ee0d1588 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:39.083011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4709949e-3897-4156-a3ad-2c89ee0d1588 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:39.085204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4709949e-3897-4156-a3ad-2c89ee0d1588 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:39.103312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e253f951fb1949408e9e966178cc782d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.103978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e253f951fb1949408e9e966178cc782d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.116832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e253f951fb1949408e9e966178cc782d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.123683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e253f951fb1949408e9e966178cc782d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.124377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e253f951fb1949408e9e966178cc782d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.206379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 744/1488] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:38 2022] DELETE /identity/v3/users/e253f951fb1949408e9e966178cc782d => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:39.209879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 745/1489] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:39 2022] POST /identity/v3/users => generated 293 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:39.211462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:39.212906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:39.277985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98600903-4e4a-4f60-8edb-5fcc87ce1e68 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:39.282858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a8b206-1c3e-4ea1-8e3d-e755673141be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:39.283504 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a8b206-1c3e-4ea1-8e3d-e755673141be admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:39.283811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a8b206-1c3e-4ea1-8e3d-e755673141be admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:39.284087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a8b206-1c3e-4ea1-8e3d-e755673141be admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:39.284732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a8b206-1c3e-4ea1-8e3d-e755673141be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:39.287638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a8b206-1c3e-4ea1-8e3d-e755673141be admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:39.297981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 745/1490] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:39 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:39.302890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4709949e-3897-4156-a3ad-2c89ee0d1588 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:39.316044 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4709949e-3897-4156-a3ad-2c89ee0d1588 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:39.330937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:39.331911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:39.331911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:39.332172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:39.333107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:39.335239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:39.349708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 746/1491] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:39 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:39.351980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a8b206-1c3e-4ea1-8e3d-e755673141be admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:39.365166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a8b206-1c3e-4ea1-8e3d-e755673141be admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:39.383171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd3ec2b-fb7a-409e-b42a-817e208b8238 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:39.383855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd3ec2b-fb7a-409e-b42a-817e208b8238 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:39.384258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd3ec2b-fb7a-409e-b42a-817e208b8238 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:39.384717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd3ec2b-fb7a-409e-b42a-817e208b8238 admin admin] PATH_INFO: `/v3/projects/8f7b5563d6884f0c83cd4f5cfbe4b0d1/users/b432e19a00fb40d69553931008195de7/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:39.404036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd3ec2b-fb7a-409e-b42a-817e208b8238 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f7b5563d6884f0c83cd4f5cfbe4b0d1, user_id=b432e19a00fb40d69553931008195de7, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:39.406805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd3ec2b-fb7a-409e-b42a-817e208b8238 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:39.499447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:39.499447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:39.511737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4f42da4-f981-446b-9b3a-6511fa7d051b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:39.539915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 746/1492] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:39 2022] PUT /identity/v3/projects/8f7b5563d6884f0c83cd4f5cfbe4b0d1/users/b432e19a00fb40d69553931008195de7/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:39.547156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:39.547156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:39.547156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:39.547156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] PATH_INFO: `/v3/projects/9dac9e5701d54dc3bc4c69462996927c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:39.550361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=9dac9e5701d54dc3bc4c69462996927c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:39.556955 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:39.560262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d22b0a-d6d9-41b4-88a3-186eaad1287f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:39.560551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d22b0a-d6d9-41b4-88a3-186eaad1287f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:39.560833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d22b0a-d6d9-41b4-88a3-186eaad1287f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:39.603712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a8d22b0a-d6d9-41b4-88a3-186eaad1287f None None] MFA Rules not processed for user `b432e19a00fb40d69553931008195de7`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:39.609980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a8d22b0a-d6d9-41b4-88a3-186eaad1287f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:39.721252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9dac9e5701d54dc3bc4c69462996927c', 'request_id': 'req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.741137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 747/1493] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:39.757372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d22b0a-d6d9-41b4-88a3-186eaad1287f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:39.759403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a8d22b0a-d6d9-41b4-88a3-186eaad1287f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:39.825640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9dac9e5701d54dc3bc4c69462996927c', 'request_id': 'req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.833417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9dac9e5701d54dc3bc4c69462996927c', 'request_id': 'req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.845923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] The token cache is being invalidate because project 9dac9e5701d54dc3bc4c69462996927c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:39.846218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.846907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:39.847252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d22b0a-d6d9-41b4-88a3-186eaad1287f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:39.849146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 747/1494] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:39 2022] DELETE /identity/v3/projects/9dac9e5701d54dc3bc4c69462996927c => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:39.853920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:39.855358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:39.911295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:39.911954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:39.912234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:39.912507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:39.914538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:39.938186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fb57f9-ad01-41ef-8d2d-9fb7058604ac admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:39.949077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:39.949736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:39.950032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:39.950275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] PATH_INFO: `/v3/users/f67337ac08f444f1ac48ce642e7938d7` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:39.953655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f67337ac08f444f1ac48ce642e7938d7)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:39.955973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:40.013367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:40.014848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:40.029136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 748/1495] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:02:40.194906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f67337ac08f444f1ac48ce642e7938d7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:40.688407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f67337ac08f444f1ac48ce642e7938d7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:40.708482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f67337ac08f444f1ac48ce642e7938d7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:40.715598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f67337ac08f444f1ac48ce642e7938d7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:40.721214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-32563a8e-8a7f-44f0-b8eb-bf9898c22570 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f67337ac08f444f1ac48ce642e7938d7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:40.723446 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 748/1496] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:39 2022] DELETE /identity/v3/users/f67337ac08f444f1ac48ce642e7938d7 => generated 0 bytes in 871 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:41.221881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:41.236216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0febda-ac85-4c90-9870-3904a0e8f68b service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:41.349984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:41.352146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:41.352650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:41.354131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] PATH_INFO: `/v3/projects/39d16d1c39f64a118a38b1a7060891af` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:41.355950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39d16d1c39f64a118a38b1a7060891af)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:41.363201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:41.560942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:41.657410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39d16d1c39f64a118a38b1a7060891af', 'request_id': 'req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:41.662286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39d16d1c39f64a118a38b1a7060891af', 'request_id': 'req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:41.668738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39d16d1c39f64a118a38b1a7060891af', 'request_id': 'req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:41.751351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] The token cache is being invalidate because project 39d16d1c39f64a118a38b1a7060891af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:41.752679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:41.754918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e844ba-fb0f-47a2-b3c6-87620ed44ee5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:41.763552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 749/1497] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:41 2022] DELETE /identity/v3/projects/39d16d1c39f64a118a38b1a7060891af => generated 0 bytes in 543 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:41.772245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974511d6-0120-492d-8209-5ffcc3060de3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:41.772245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974511d6-0120-492d-8209-5ffcc3060de3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:41.772245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974511d6-0120-492d-8209-5ffcc3060de3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:41.810865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-974511d6-0120-492d-8209-5ffcc3060de3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:41.812739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-974511d6-0120-492d-8209-5ffcc3060de3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:41.931440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 749/1498] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:41.933131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8513ea9-6150-4172-b49e-e37c1cc1dbba None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:41.933431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8513ea9-6150-4172-b49e-e37c1cc1dbba None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:41.933703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8513ea9-6150-4172-b49e-e37c1cc1dbba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:42.005520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a8513ea9-6150-4172-b49e-e37c1cc1dbba None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:42.005520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a8513ea9-6150-4172-b49e-e37c1cc1dbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:42.005520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 750/1499] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:02:42.005776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974511d6-0120-492d-8209-5ffcc3060de3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:42.005776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974511d6-0120-492d-8209-5ffcc3060de3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:42.005776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:42.005776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:42.006116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:42.006116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:42.006116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:42.006116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:42.006116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:42.008754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 750/1500] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:42.027517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8513ea9-6150-4172-b49e-e37c1cc1dbba None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:42.039113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8513ea9-6150-4172-b49e-e37c1cc1dbba None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:42.044827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64d00e-e022-4cdd-b7dc-17a10ac7b4dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:42.045046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64d00e-e022-4cdd-b7dc-17a10ac7b4dc admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:42.045330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64d00e-e022-4cdd-b7dc-17a10ac7b4dc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:42.045605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64d00e-e022-4cdd-b7dc-17a10ac7b4dc admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:42.046460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af64d00e-e022-4cdd-b7dc-17a10ac7b4dc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:42.048456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af64d00e-e022-4cdd-b7dc-17a10ac7b4dc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:42.253256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 751/1501] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:42 2022] POST /identity/v3/projects => generated 377 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:42.258206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:42.268077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4b3f7f-e0b2-4559-b8c4-70a1023588c2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:42.324949 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c02564-3569-45ec-99fc-f4dbd6dae1b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:42.325566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c02564-3569-45ec-99fc-f4dbd6dae1b5 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:42.325848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c02564-3569-45ec-99fc-f4dbd6dae1b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:42.326124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c02564-3569-45ec-99fc-f4dbd6dae1b5 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:42.326766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c02564-3569-45ec-99fc-f4dbd6dae1b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:42.330356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c02564-3569-45ec-99fc-f4dbd6dae1b5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:42.481241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 751/1502] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:42 2022] POST /identity/v3/users => generated 289 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:42.483001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64d00e-e022-4cdd-b7dc-17a10ac7b4dc admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:42.500278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64d00e-e022-4cdd-b7dc-17a10ac7b4dc admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:42.511256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605c68d1-71b5-4e11-bfc2-d2991381aa4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:42.511832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605c68d1-71b5-4e11-bfc2-d2991381aa4c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:42.512117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605c68d1-71b5-4e11-bfc2-d2991381aa4c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:42.512464 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605c68d1-71b5-4e11-bfc2-d2991381aa4c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:42.513112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605c68d1-71b5-4e11-bfc2-d2991381aa4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:42.514706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605c68d1-71b5-4e11-bfc2-d2991381aa4c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:42.528518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 752/1503] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:42 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:42.532437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c02564-3569-45ec-99fc-f4dbd6dae1b5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:42.544214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c02564-3569-45ec-99fc-f4dbd6dae1b5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:42.549475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1117b9df-084a-4f87-b6f7-487c4b94c786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:42.550319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117b9df-084a-4f87-b6f7-487c4b94c786 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:42.563118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117b9df-084a-4f87-b6f7-487c4b94c786 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:42.563118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1117b9df-084a-4f87-b6f7-487c4b94c786 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:42.563118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117b9df-084a-4f87-b6f7-487c4b94c786 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:42.563118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1117b9df-084a-4f87-b6f7-487c4b94c786 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:42.565481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 752/1504] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:42 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:42.570024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605c68d1-71b5-4e11-bfc2-d2991381aa4c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:42.584773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605c68d1-71b5-4e11-bfc2-d2991381aa4c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:42.586810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:42.594444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c25d9d-4ad7-45fe-b623-d258bcf4114b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:42.599184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c25d9d-4ad7-45fe-b623-d258bcf4114b admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:42.600519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c25d9d-4ad7-45fe-b623-d258bcf4114b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:42.601350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c25d9d-4ad7-45fe-b623-d258bcf4114b admin admin] PATH_INFO: `/v3/projects/9d1926d34e824016927d5d2eeefca945/users/5deec494f31646fb93b3e66e32def88a/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:42.629746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c25d9d-4ad7-45fe-b623-d258bcf4114b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d1926d34e824016927d5d2eeefca945, user_id=5deec494f31646fb93b3e66e32def88a, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:42.633223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c25d9d-4ad7-45fe-b623-d258bcf4114b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:42.723801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 753/1505] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:42 2022] PUT /identity/v3/projects/9d1926d34e824016927d5d2eeefca945/users/5deec494f31646fb93b3e66e32def88a/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:42.735607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e87702-50be-44bd-888f-3b1bd96c3f64 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:42.736093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e87702-50be-44bd-888f-3b1bd96c3f64 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:42.736541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e87702-50be-44bd-888f-3b1bd96c3f64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:42.784288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-49e87702-50be-44bd-888f-3b1bd96c3f64 None None] MFA Rules not processed for user `5deec494f31646fb93b3e66e32def88a`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:42.784928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-49e87702-50be-44bd-888f-3b1bd96c3f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:42.909558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 753/1506] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:42 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:42.914810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c25d9d-4ad7-45fe-b623-d258bcf4114b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:43.008139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c25d9d-4ad7-45fe-b623-d258bcf4114b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:43.013381 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021aca5e-5829-48e7-a18a-0e78ad415235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:43.013997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021aca5e-5829-48e7-a18a-0e78ad415235 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:43.014358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021aca5e-5829-48e7-a18a-0e78ad415235 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:43.014675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-021aca5e-5829-48e7-a18a-0e78ad415235 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:43.015417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021aca5e-5829-48e7-a18a-0e78ad415235 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:43.017384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-021aca5e-5829-48e7-a18a-0e78ad415235 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:43.116353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 754/1507] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:42 2022] POST /identity/v3/projects => generated 377 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:43.122019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e87702-50be-44bd-888f-3b1bd96c3f64 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:43.131679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e87702-50be-44bd-888f-3b1bd96c3f64 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:43.186427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f739579-324f-4bcb-8db0-5bb46cad0c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:43.187060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f739579-324f-4bcb-8db0-5bb46cad0c9d admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:43.187336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f739579-324f-4bcb-8db0-5bb46cad0c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:43.187603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f739579-324f-4bcb-8db0-5bb46cad0c9d admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:43.188252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f739579-324f-4bcb-8db0-5bb46cad0c9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:43.190164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f739579-324f-4bcb-8db0-5bb46cad0c9d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:43.209178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:43.317925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 754/1508] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:43 2022] POST /identity/v3/users => generated 295 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:43.321891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021aca5e-5829-48e7-a18a-0e78ad415235 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:43.332009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-021aca5e-5829-48e7-a18a-0e78ad415235 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:43.352288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729e7ece-45b0-40e2-b8d3-a7a45aaa9fd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:43.352288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e7ece-45b0-40e2-b8d3-a7a45aaa9fd8 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:43.352288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e7ece-45b0-40e2-b8d3-a7a45aaa9fd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:43.352632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729e7ece-45b0-40e2-b8d3-a7a45aaa9fd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:43.353137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729e7ece-45b0-40e2-b8d3-a7a45aaa9fd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:43.355625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729e7ece-45b0-40e2-b8d3-a7a45aaa9fd8 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:43.373375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 755/1509] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:43 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:43.374648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f739579-324f-4bcb-8db0-5bb46cad0c9d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:43.386478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f739579-324f-4bcb-8db0-5bb46cad0c9d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:43.390131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e329b0-5b67-40d4-8573-a8bb5c35cee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:43.390783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e329b0-5b67-40d4-8573-a8bb5c35cee6 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:43.391096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e329b0-5b67-40d4-8573-a8bb5c35cee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:43.391367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e329b0-5b67-40d4-8573-a8bb5c35cee6 admin admin] PATH_INFO: `/v3/projects/24ba12241be849b19b350c3755c6abd6/users/f125bfe20cbf4b97b2ecdbc1198daa5d/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:43.407581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e329b0-5b67-40d4-8573-a8bb5c35cee6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24ba12241be849b19b350c3755c6abd6, user_id=f125bfe20cbf4b97b2ecdbc1198daa5d, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:43.410311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e329b0-5b67-40d4-8573-a8bb5c35cee6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:43.506130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 755/1510] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:43 2022] PUT /identity/v3/projects/24ba12241be849b19b350c3755c6abd6/users/f125bfe20cbf4b97b2ecdbc1198daa5d/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:43.510116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729e7ece-45b0-40e2-b8d3-a7a45aaa9fd8 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:43.520362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729e7ece-45b0-40e2-b8d3-a7a45aaa9fd8 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:43.577158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b2f210-4e1a-4e36-98a0-23a2cb92d054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:43.577697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b2f210-4e1a-4e36-98a0-23a2cb92d054 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:43.577973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b2f210-4e1a-4e36-98a0-23a2cb92d054 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:43.578242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b2f210-4e1a-4e36-98a0-23a2cb92d054 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:43.578879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b2f210-4e1a-4e36-98a0-23a2cb92d054 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:43.580848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b2f210-4e1a-4e36-98a0-23a2cb92d054 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:43.590374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 756/1511] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:43 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:43.594514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e329b0-5b67-40d4-8573-a8bb5c35cee6 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:43.604946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e329b0-5b67-40d4-8573-a8bb5c35cee6 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:43.609547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:43.610185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:43.610462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:43.610735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] PATH_INFO: `/v3/projects/24ba12241be849b19b350c3755c6abd6/users/f125bfe20cbf4b97b2ecdbc1198daa5d/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:43.612685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24ba12241be849b19b350c3755c6abd6, user_id=f125bfe20cbf4b97b2ecdbc1198daa5d, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:43.615861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:43.625814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 756/1512] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:43 2022] PUT /identity/v3/projects/24ba12241be849b19b350c3755c6abd6/users/f125bfe20cbf4b97b2ecdbc1198daa5d/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:43.636624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19de0300-0455-49a9-9799-eb9f962020e0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:43.636914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19de0300-0455-49a9-9799-eb9f962020e0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:43.655066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19de0300-0455-49a9-9799-eb9f962020e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:43.665605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-19de0300-0455-49a9-9799-eb9f962020e0 None None] MFA Rules not processed for user `f125bfe20cbf4b97b2ecdbc1198daa5d`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:43.666839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-19de0300-0455-49a9-9799-eb9f962020e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:43.735502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 757/1513] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:43 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:43.746754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:43.748489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:43.811260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5109e1-cb1e-4e6e-bcae-60c890cd69e9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:43.814912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18666ad6-12d8-4f63-bc17-c4987f595691 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:43.815542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18666ad6-12d8-4f63-bc17-c4987f595691 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:43.815815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18666ad6-12d8-4f63-bc17-c4987f595691 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:43.816081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18666ad6-12d8-4f63-bc17-c4987f595691 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:43.873645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18666ad6-12d8-4f63-bc17-c4987f595691 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:43.875167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18666ad6-12d8-4f63-bc17-c4987f595691 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:43.884094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 757/1514] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:43 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:44.477285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:44.556893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:44.911345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:45.117538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:45.379733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:45.629568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:45.831342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:46.035188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:46.058001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:46.458530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:46.843595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:47.890064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7efca3-0ebd-422b-90ee-d8450cdced08 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:47.891355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7efca3-0ebd-422b-90ee-d8450cdced08 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:47.892340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7efca3-0ebd-422b-90ee-d8450cdced08 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:47.894796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 758/1515] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:02:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:02:47.903509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf160852-a095-48dd-a614-1fa906882112 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:47.903509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf160852-a095-48dd-a614-1fa906882112 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:47.904048 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf160852-a095-48dd-a614-1fa906882112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:47.931423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-bf160852-a095-48dd-a614-1fa906882112 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:47.933142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-bf160852-a095-48dd-a614-1fa906882112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:48.034451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 758/1516] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:02:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:48.079993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7efca3-0ebd-422b-90ee-d8450cdced08 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:48.081942 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ae7efca3-0ebd-422b-90ee-d8450cdced08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:48.108860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7efca3-0ebd-422b-90ee-d8450cdced08 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:48.115500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c389d0f3-41b6-4d83-8f0e-9c7499f66593 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:48.116435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389d0f3-41b6-4d83-8f0e-9c7499f66593 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:48.116435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389d0f3-41b6-4d83-8f0e-9c7499f66593 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:48.116664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c389d0f3-41b6-4d83-8f0e-9c7499f66593 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:48.131510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c389d0f3-41b6-4d83-8f0e-9c7499f66593 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:48.133121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c389d0f3-41b6-4d83-8f0e-9c7499f66593 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:48.145641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 759/1517] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:48.156856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:48.371144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf160852-a095-48dd-a614-1fa906882112 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:48.372587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-bf160852-a095-48dd-a614-1fa906882112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:48.441827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf160852-a095-48dd-a614-1fa906882112 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:48.444673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:48.445306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:48.445586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:48.445858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] PATH_INFO: `/v3/users/d7130b0645074709a7c5c7373b162cab` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:48.447001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7130b0645074709a7c5c7373b162cab)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:48.448966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:48.516870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:48.673872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7130b0645074709a7c5c7373b162cab'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:48.750653 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7130b0645074709a7c5c7373b162cab'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:48.763610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7130b0645074709a7c5c7373b162cab'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:48.778199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7130b0645074709a7c5c7373b162cab'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:48.778973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7130b0645074709a7c5c7373b162cab'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:48.789076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 759/1518] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:48 2022] DELETE /identity/v3/users/d7130b0645074709a7c5c7373b162cab => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:48.792399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65064d51-fbe0-4111-acac-1087f45eb8bd None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:48.792685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65064d51-fbe0-4111-acac-1087f45eb8bd None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:48.793020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65064d51-fbe0-4111-acac-1087f45eb8bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:48.823199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-65064d51-fbe0-4111-acac-1087f45eb8bd None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:48.825022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-65064d51-fbe0-4111-acac-1087f45eb8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:48.915157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 760/1519] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:48.924942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:48.926943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:48.983638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b416422-12ba-497b-828f-5db81e83f305 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:48.987241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff6a282-faf0-435f-b16b-8b4dd87b1955 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:48.987847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff6a282-faf0-435f-b16b-8b4dd87b1955 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:48.988130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff6a282-faf0-435f-b16b-8b4dd87b1955 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:48.988405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff6a282-faf0-435f-b16b-8b4dd87b1955 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:48.998112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff6a282-faf0-435f-b16b-8b4dd87b1955 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:48.999600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff6a282-faf0-435f-b16b-8b4dd87b1955 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:49.010154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 760/1520] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:49.404239 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:49.483977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65064d51-fbe0-4111-acac-1087f45eb8bd None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:49.485410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-65064d51-fbe0-4111-acac-1087f45eb8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:49.497880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65064d51-fbe0-4111-acac-1087f45eb8bd None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:49.502803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:49.503472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:49.503810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:49.509768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] PATH_INFO: `/v3/projects/b61360ac8eff44f18d6d5425db0a276f` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:49.509768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b61360ac8eff44f18d6d5425db0a276f)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:49.509768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:49.619839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b61360ac8eff44f18d6d5425db0a276f', 'request_id': 'req-0247faf6-68d9-463c-92f1-250cd79135f0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:49.626023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b61360ac8eff44f18d6d5425db0a276f', 'request_id': 'req-0247faf6-68d9-463c-92f1-250cd79135f0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:49.632082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b61360ac8eff44f18d6d5425db0a276f', 'request_id': 'req-0247faf6-68d9-463c-92f1-250cd79135f0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:49.648120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:49.681633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] The token cache is being invalidate because project b61360ac8eff44f18d6d5425db0a276f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:49.681920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:49.682550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:49.684119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 761/1521] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:49 2022] DELETE /identity/v3/projects/b61360ac8eff44f18d6d5425db0a276f => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:49.687997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff6a282-faf0-435f-b16b-8b4dd87b1955 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:49.698034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff6a282-faf0-435f-b16b-8b4dd87b1955 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:49.754042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:49.755284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:49.755562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:49.755833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] PATH_INFO: `/v3/users/b2238e7f727145d3998b048a367297c0` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:49.757004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2238e7f727145d3998b048a367297c0)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:49.758930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:49.941669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2238e7f727145d3998b048a367297c0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.037276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2238e7f727145d3998b048a367297c0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.055210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2238e7f727145d3998b048a367297c0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.074475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2238e7f727145d3998b048a367297c0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.075453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-910fa8c2-c45a-4f93-ba17-1cd174146ce5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2238e7f727145d3998b048a367297c0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.082095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 761/1522] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:49 2022] DELETE /identity/v3/users/b2238e7f727145d3998b048a367297c0 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:50.328675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:50.391347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:50.587108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:50.587816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:50.589266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:50.603215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247faf6-68d9-463c-92f1-250cd79135f0 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:50.642733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:50.659821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:50.660467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:50.661288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:50.661568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] PATH_INFO: `/v3/projects/575422ec43664a49a38036f433a91bc8` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:50.662703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=575422ec43664a49a38036f433a91bc8)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:50.664653 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:50.768652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '575422ec43664a49a38036f433a91bc8', 'request_id': 'req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.775366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '575422ec43664a49a38036f433a91bc8', 'request_id': 'req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.781640 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '575422ec43664a49a38036f433a91bc8', 'request_id': 'req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.873637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] The token cache is being invalidate because project 575422ec43664a49a38036f433a91bc8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:50.874048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.875138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-e23eab38-dd5e-4d6e-9f2e-652833a68cd5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:50.877752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 762/1523] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:50 2022] DELETE /identity/v3/projects/575422ec43664a49a38036f433a91bc8 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:50.889153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5835dec-ef76-4d19-ae93-9555da0efda0 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:50.889921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5835dec-ef76-4d19-ae93-9555da0efda0 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:50.890428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5835dec-ef76-4d19-ae93-9555da0efda0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:50.901745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:50.950512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a5835dec-ef76-4d19-ae93-9555da0efda0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:50.953721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a5835dec-ef76-4d19-ae93-9555da0efda0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:51.010709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:51.089495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 762/1524] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:51.100645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ec6d5b-4d75-4b2b-ae2b-26418a50262d None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.103724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ec6d5b-4d75-4b2b-ae2b-26418a50262d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.104084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ec6d5b-4d75-4b2b-ae2b-26418a50262d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.151502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-89ec6d5b-4d75-4b2b-ae2b-26418a50262d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:51.152484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-89ec6d5b-4d75-4b2b-ae2b-26418a50262d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:51.184778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 763/1525] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:51.192408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5835dec-ef76-4d19-ae93-9555da0efda0 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:51.217316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5835dec-ef76-4d19-ae93-9555da0efda0 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:51.239625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:51.240386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.240776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.241279 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.242053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:51.242461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:51.244435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:51.259754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 763/1526] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:51.264310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ec6d5b-4d75-4b2b-ae2b-26418a50262d None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:51.276812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ec6d5b-4d75-4b2b-ae2b-26418a50262d None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:51.284065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:51.285081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.285081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.285262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.286185 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:51.288391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:51.408157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 764/1527] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:51 2022] POST /identity/v3/projects => generated 363 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:51.413006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:51.425682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485ab32b-0026-46c9-a9a7-54e103481d98 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:51.494583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:51.501195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:51.503338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:51.514749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4029b4-2514-4db0-8e37-52ce788b348d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:51.516107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4029b4-2514-4db0-8e37-52ce788b348d admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.517660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4029b4-2514-4db0-8e37-52ce788b348d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.518288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4029b4-2514-4db0-8e37-52ce788b348d admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.519335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4029b4-2514-4db0-8e37-52ce788b348d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:51.522395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4029b4-2514-4db0-8e37-52ce788b348d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:51.609179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 764/1528] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:51 2022] POST /identity/v3/users => generated 282 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:51.614219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4029b4-2514-4db0-8e37-52ce788b348d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:51.632817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b097ac-7d41-49b4-9097-d9624cc95a96 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:51.642889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4029b4-2514-4db0-8e37-52ce788b348d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:51.648693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09296973-fec7-44fa-b950-df2513f0ca56 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:51.650489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09296973-fec7-44fa-b950-df2513f0ca56 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.652088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09296973-fec7-44fa-b950-df2513f0ca56 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.652535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09296973-fec7-44fa-b950-df2513f0ca56 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.656744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-09296973-fec7-44fa-b950-df2513f0ca56 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:51.657585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faf0aee-21bb-4b18-b209-80f09acf79e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:51.658907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faf0aee-21bb-4b18-b209-80f09acf79e4 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.661044 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faf0aee-21bb-4b18-b209-80f09acf79e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.661476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faf0aee-21bb-4b18-b209-80f09acf79e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.665176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faf0aee-21bb-4b18-b209-80f09acf79e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:51.669587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faf0aee-21bb-4b18-b209-80f09acf79e4 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:51.687118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 765/1529] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:51.693525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faf0aee-21bb-4b18-b209-80f09acf79e4 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:51.710619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faf0aee-21bb-4b18-b209-80f09acf79e4 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:51.720773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a57f13-8cda-4000-a00a-7b234e918620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:51.721552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a57f13-8cda-4000-a00a-7b234e918620 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.721837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a57f13-8cda-4000-a00a-7b234e918620 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.722169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a57f13-8cda-4000-a00a-7b234e918620 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.723534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a57f13-8cda-4000-a00a-7b234e918620 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:51.725782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a57f13-8cda-4000-a00a-7b234e918620 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:51.747626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:51.748534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 766/1530] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:51 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:51.749437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a57f13-8cda-4000-a00a-7b234e918620 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:51.768745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a57f13-8cda-4000-a00a-7b234e918620 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:51.773400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09296973-fec7-44fa-b950-df2513f0ca56 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:51.775256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09296973-fec7-44fa-b950-df2513f0ca56 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:51.775788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6e42f3-c0ca-42fa-a0c8-3b3d6cbc7cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:51.776677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6e42f3-c0ca-42fa-a0c8-3b3d6cbc7cb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.777146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6e42f3-c0ca-42fa-a0c8-3b3d6cbc7cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.777483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d6e42f3-c0ca-42fa-a0c8-3b3d6cbc7cb2 admin admin] PATH_INFO: `/v3/projects/d6acb1c06c264bb1ab14512f62fcd1fd/users/e4ad0c19dcd24f9a84d76b57d378cdfe/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.802227 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6e42f3-c0ca-42fa-a0c8-3b3d6cbc7cb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d6acb1c06c264bb1ab14512f62fcd1fd, user_id=e4ad0c19dcd24f9a84d76b57d378cdfe, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:51.802995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 765/1531] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:51 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:51.805600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d6e42f3-c0ca-42fa-a0c8-3b3d6cbc7cb2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:51.907523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 767/1532] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:51 2022] PUT /identity/v3/projects/d6acb1c06c264bb1ab14512f62fcd1fd/users/e4ad0c19dcd24f9a84d76b57d378cdfe/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:51.929992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39386c5-de21-4894-a598-9e4e365fc2e6 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:51.930280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39386c5-de21-4894-a598-9e4e365fc2e6 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:51.930541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39386c5-de21-4894-a598-9e4e365fc2e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:51.968999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c39386c5-de21-4894-a598-9e4e365fc2e6 None None] MFA Rules not processed for user `e4ad0c19dcd24f9a84d76b57d378cdfe`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:51.970385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c39386c5-de21-4894-a598-9e4e365fc2e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:52.084198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 766/1533] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:51 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:52.089459 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6e42f3-c0ca-42fa-a0c8-3b3d6cbc7cb2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:52.248292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d6e42f3-c0ca-42fa-a0c8-3b3d6cbc7cb2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:52.260409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:52.261595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:52.262026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:52.267058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:52.267058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:52.267058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:52.469586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 768/1534] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:52 2022] POST /identity/v3/projects => generated 363 bytes in 381 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:52.473936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39386c5-de21-4894-a598-9e4e365fc2e6 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:52.491772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39386c5-de21-4894-a598-9e4e365fc2e6 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:52.592580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856f6dd3-f3ea-42db-bc3f-0d241fba56d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:52.593198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856f6dd3-f3ea-42db-bc3f-0d241fba56d7 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:52.593472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856f6dd3-f3ea-42db-bc3f-0d241fba56d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:52.593738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856f6dd3-f3ea-42db-bc3f-0d241fba56d7 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:52.594367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856f6dd3-f3ea-42db-bc3f-0d241fba56d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:52.597260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856f6dd3-f3ea-42db-bc3f-0d241fba56d7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:52.722882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:52.725588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:52.728557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 767/1535] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:52 2022] POST /identity/v3/users => generated 288 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:52.729944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856f6dd3-f3ea-42db-bc3f-0d241fba56d7 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:52.743492 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856f6dd3-f3ea-42db-bc3f-0d241fba56d7 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:52.746812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ace8e5e-7066-4cdc-ab90-0869d4c253aa admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:52.749767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e333cde-b607-4bd4-8025-8ba42f49da9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:52.750825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e333cde-b607-4bd4-8025-8ba42f49da9e admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:52.751359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e333cde-b607-4bd4-8025-8ba42f49da9e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:52.752209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e333cde-b607-4bd4-8025-8ba42f49da9e admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:52.755311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e333cde-b607-4bd4-8025-8ba42f49da9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:52.757510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e333cde-b607-4bd4-8025-8ba42f49da9e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:52.766000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:52.766778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:52.767188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:52.767558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] PATH_INFO: `/v3/users/b432e19a00fb40d69553931008195de7` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:52.769600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] RBAC: Authorizing `identity:delete_user(user_id=b432e19a00fb40d69553931008195de7)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:52.772268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:52.773015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 768/1536] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:52 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:52.777752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e333cde-b607-4bd4-8025-8ba42f49da9e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:52.795080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e333cde-b607-4bd4-8025-8ba42f49da9e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:52.798590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7f5b0a-d58d-43b2-98dd-b63cf0073337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:52.799489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f5b0a-d58d-43b2-98dd-b63cf0073337 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:52.800104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f5b0a-d58d-43b2-98dd-b63cf0073337 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:52.800772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f5b0a-d58d-43b2-98dd-b63cf0073337 admin admin] PATH_INFO: `/v3/projects/6876cbdeea004273ba55aa6b16ed73a5/users/ba21641b2bab46058a9e22107b5581d3/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:52.832183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7f5b0a-d58d-43b2-98dd-b63cf0073337 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6876cbdeea004273ba55aa6b16ed73a5, user_id=ba21641b2bab46058a9e22107b5581d3, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:52.836544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c7f5b0a-d58d-43b2-98dd-b63cf0073337 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:52.921520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 769/1537] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:52 2022] PUT /identity/v3/projects/6876cbdeea004273ba55aa6b16ed73a5/users/ba21641b2bab46058a9e22107b5581d3/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:52.925666 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7f5b0a-d58d-43b2-98dd-b63cf0073337 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:53.007479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b432e19a00fb40d69553931008195de7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.008927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7f5b0a-d58d-43b2-98dd-b63cf0073337 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:53.014438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839d0f5-dfc0-4649-80a3-659a5ec7ad68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:53.015081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839d0f5-dfc0-4649-80a3-659a5ec7ad68 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.015565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839d0f5-dfc0-4649-80a3-659a5ec7ad68 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.015841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e839d0f5-dfc0-4649-80a3-659a5ec7ad68 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.016490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e839d0f5-dfc0-4649-80a3-659a5ec7ad68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:53.018088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e839d0f5-dfc0-4649-80a3-659a5ec7ad68 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:53.029095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 770/1538] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:52 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:53.034071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839d0f5-dfc0-4649-80a3-659a5ec7ad68 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:53.048457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e839d0f5-dfc0-4649-80a3-659a5ec7ad68 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:53.054971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587722cc-58ae-4cd1-b96f-86c84bd98d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:53.055745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587722cc-58ae-4cd1-b96f-86c84bd98d73 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.056079 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587722cc-58ae-4cd1-b96f-86c84bd98d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.056577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587722cc-58ae-4cd1-b96f-86c84bd98d73 admin admin] PATH_INFO: `/v3/projects/6876cbdeea004273ba55aa6b16ed73a5/users/ba21641b2bab46058a9e22107b5581d3/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.058584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587722cc-58ae-4cd1-b96f-86c84bd98d73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6876cbdeea004273ba55aa6b16ed73a5, user_id=ba21641b2bab46058a9e22107b5581d3, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:53.061379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587722cc-58ae-4cd1-b96f-86c84bd98d73 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:53.077230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 771/1539] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:53 2022] PUT /identity/v3/projects/6876cbdeea004273ba55aa6b16ed73a5/users/ba21641b2bab46058a9e22107b5581d3/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:53.086679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aec293d-5c92-4672-9073-d33604b70e0e None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.087032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aec293d-5c92-4672-9073-d33604b70e0e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.087273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aec293d-5c92-4672-9073-d33604b70e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.111078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b432e19a00fb40d69553931008195de7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.119848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6aec293d-5c92-4672-9073-d33604b70e0e None None] MFA Rules not processed for user `ba21641b2bab46058a9e22107b5581d3`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:53.121138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6aec293d-5c92-4672-9073-d33604b70e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:53.138061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b432e19a00fb40d69553931008195de7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.148525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b432e19a00fb40d69553931008195de7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.149565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b432e19a00fb40d69553931008195de7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.157379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 769/1540] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:52 2022] DELETE /identity/v3/users/b432e19a00fb40d69553931008195de7 => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:53.158368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:53.160113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:53.168252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:53.217367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 772/1541] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:53 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:02:53.220118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af81d528-3b6d-440b-b7c5-226cfee0d0b8 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.220577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af81d528-3b6d-440b-b7c5-226cfee0d0b8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.220984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af81d528-3b6d-440b-b7c5-226cfee0d0b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.243030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b228f0a-6559-421a-b5e0-01ddf32a22be admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:53.244587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-af81d528-3b6d-440b-b7c5-226cfee0d0b8 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:53.245908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-af81d528-3b6d-440b-b7c5-226cfee0d0b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:53.248250 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:53.248944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.249221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.249491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] PATH_INFO: `/v3/users/d40a61d7590f49a09cfdfc4e4dedacae` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.250620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] RBAC: Authorizing `identity:delete_user(user_id=d40a61d7590f49a09cfdfc4e4dedacae)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:53.252709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:53.256829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 773/1542] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:53.257649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af81d528-3b6d-440b-b7c5-226cfee0d0b8 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:53.258920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-af81d528-3b6d-440b-b7c5-226cfee0d0b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:53.273077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=4,keystone:SELECT=86,keystone:DELETE=10 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:53.315924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af81d528-3b6d-440b-b7c5-226cfee0d0b8 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:53.320231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:53.320841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.321121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.321397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.382158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:53.383637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:53.393552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 774/1543] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:53 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:53.394467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:53.396329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:53.457891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd1b4c0-8439-4183-b718-ae404ccd0974 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:53.462035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd40a61d7590f49a09cfdfc4e4dedacae'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.462173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a34575a-b0c7-45f7-8abb-d59d4684535f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:53.462706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a34575a-b0c7-45f7-8abb-d59d4684535f service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.463010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a34575a-b0c7-45f7-8abb-d59d4684535f service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.463286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a34575a-b0c7-45f7-8abb-d59d4684535f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.472509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a34575a-b0c7-45f7-8abb-d59d4684535f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:53.473951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a34575a-b0c7-45f7-8abb-d59d4684535f service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:53.483376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 775/1544] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:53.556984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd40a61d7590f49a09cfdfc4e4dedacae'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.576997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd40a61d7590f49a09cfdfc4e4dedacae'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.587152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd40a61d7590f49a09cfdfc4e4dedacae'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.590269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd40a61d7590f49a09cfdfc4e4dedacae'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:53.594941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 770/1545] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:53 2022] DELETE /identity/v3/users/d40a61d7590f49a09cfdfc4e4dedacae => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:53.600002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c762111c-2bea-4ea6-a015-81c14d6e98ed None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.600361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c762111c-2bea-4ea6-a015-81c14d6e98ed None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.600757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c762111c-2bea-4ea6-a015-81c14d6e98ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.620515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c762111c-2bea-4ea6-a015-81c14d6e98ed None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:53.621819 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c762111c-2bea-4ea6-a015-81c14d6e98ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:53.682374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 776/1546] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:53.694905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:53.696635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:53.789315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e6149bf-e0e5-4e28-b9c4-b0f8a85cacec admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:53.794484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:53.795408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:53.795746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:53.796139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:53.807492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:53.809207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:53.820222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 771/1547] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:54.226696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c762111c-2bea-4ea6-a015-81c14d6e98ed None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:54.228537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c762111c-2bea-4ea6-a015-81c14d6e98ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:54.243063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c762111c-2bea-4ea6-a015-81c14d6e98ed None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:54.250376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:54.251065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:54.251482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:54.251764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] PATH_INFO: `/v3/projects/8f7b5563d6884f0c83cd4f5cfbe4b0d1` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:54.253583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f7b5563d6884f0c83cd4f5cfbe4b0d1)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:54.256111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:54.433146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:54.436201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:54.447595 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f7b5563d6884f0c83cd4f5cfbe4b0d1', 'request_id': 'req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.454342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f7b5563d6884f0c83cd4f5cfbe4b0d1', 'request_id': 'req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.456036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c850310-e45f-4d6f-8000-9cb7da1fec18 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:54.461435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f7b5563d6884f0c83cd4f5cfbe4b0d1', 'request_id': 'req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.523033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:54.523033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:54.523033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:54.523033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] PATH_INFO: `/v3/users/e4ad0c19dcd24f9a84d76b57d378cdfe` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:54.524177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4ad0c19dcd24f9a84d76b57d378cdfe)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:54.526175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:54.561836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] The token cache is being invalidate because project 8f7b5563d6884f0c83cd4f5cfbe4b0d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:54.562353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.562755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a2e1ff5-cf3f-4a1f-8e2f-009dca84ff3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.564302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 777/1548] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:54 2022] DELETE /identity/v3/projects/8f7b5563d6884f0c83cd4f5cfbe4b0d1 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:54.569840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3869f1-bfdb-4472-addb-1967b88e1a09 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:54.570126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3869f1-bfdb-4472-addb-1967b88e1a09 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:54.570401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3869f1-bfdb-4472-addb-1967b88e1a09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:54.590196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ad3869f1-bfdb-4472-addb-1967b88e1a09 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:54.591427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ad3869f1-bfdb-4472-addb-1967b88e1a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:54.599200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 778/1549] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:02:54.606367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0a4a17-6096-4d65-817d-08f30a606122 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:54.606687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0a4a17-6096-4d65-817d-08f30a606122 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:54.606983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0a4a17-6096-4d65-817d-08f30a606122 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:54.630373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-8e0a4a17-6096-4d65-817d-08f30a606122 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:54.631630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8e0a4a17-6096-4d65-817d-08f30a606122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:54.641360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 779/1550] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:02:54.641360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0a4a17-6096-4d65-817d-08f30a606122 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:54.642569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8e0a4a17-6096-4d65-817d-08f30a606122 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:54.654829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0a4a17-6096-4d65-817d-08f30a606122 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:54.659411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:54.660011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:54.660284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:54.660575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] PATH_INFO: `/v3/projects/02998fb8bdbf497ea6e815fee827bac9` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:54.661687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] RBAC: Authorizing `identity:delete_project(project_id=02998fb8bdbf497ea6e815fee827bac9)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:54.663658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:54.763896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4ad0c19dcd24f9a84d76b57d378cdfe'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.810553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:54.814283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:54.842447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4ad0c19dcd24f9a84d76b57d378cdfe'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.854887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4ad0c19dcd24f9a84d76b57d378cdfe'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.861588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4ad0c19dcd24f9a84d76b57d378cdfe'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.862220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4ad0c19dcd24f9a84d76b57d378cdfe'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.868742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 772/1551] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:54 2022] DELETE /identity/v3/users/e4ad0c19dcd24f9a84d76b57d378cdfe => generated 0 bytes in 436 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:54.869603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:54.870904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:54.909880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02998fb8bdbf497ea6e815fee827bac9', 'request_id': 'req-9a90b408-3d3b-47ed-99ed-f9830313780a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.917742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02998fb8bdbf497ea6e815fee827bac9', 'request_id': 'req-9a90b408-3d3b-47ed-99ed-f9830313780a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.925175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02998fb8bdbf497ea6e815fee827bac9', 'request_id': 'req-9a90b408-3d3b-47ed-99ed-f9830313780a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:54.941596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afdf37-2dac-4fb6-98d2-58ee14f66919 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:54.998504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6784009-fb6c-4009-819f-9458728d19fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:54.999148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6784009-fb6c-4009-819f-9458728d19fc admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:54.999433 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6784009-fb6c-4009-819f-9458728d19fc admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:54.999706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6784009-fb6c-4009-819f-9458728d19fc admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.000360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6784009-fb6c-4009-819f-9458728d19fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:55.000662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6784009-fb6c-4009-819f-9458728d19fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.002274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6784009-fb6c-4009-819f-9458728d19fc admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.020157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 773/1552] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:55.020157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e7b4e8-060d-4836-a942-1f4af0f4d339 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.020157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e7b4e8-060d-4836-a942-1f4af0f4d339 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.020157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e7b4e8-060d-4836-a942-1f4af0f4d339 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.023298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] The token cache is being invalidate because project 02998fb8bdbf497ea6e815fee827bac9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:55.023705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:55.024418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:55.026195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 780/1553] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:54 2022] DELETE /identity/v3/projects/02998fb8bdbf497ea6e815fee827bac9 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:55.027171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.028548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:55.042744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-26e7b4e8-060d-4836-a942-1f4af0f4d339 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:55.042981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a90b408-3d3b-47ed-99ed-f9830313780a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.043981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-26e7b4e8-060d-4836-a942-1f4af0f4d339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:55.047778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.048608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.049072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.049750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.050582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.055748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 774/1554] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:55.055748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e7b4e8-060d-4836-a942-1f4af0f4d339 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.055748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-26e7b4e8-060d-4836-a942-1f4af0f4d339 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:55.057676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.074028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e7b4e8-060d-4836-a942-1f4af0f4d339 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.078669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.079419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.079742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.080005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] PATH_INFO: `/v3/users/2f489f2d8fd54dd6870c05b4d4e9b387` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.081169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f489f2d8fd54dd6870c05b4d4e9b387)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.083138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.144586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 781/1555] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:55 2022] POST /identity/v3/projects => generated 361 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:55.145436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.147172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:55.206449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bb0f9a-0e55-410a-8cb4-fc68ce410af8 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.210021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a2c44d-1b8f-41bf-ab15-269b399b4147 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.210611 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a2c44d-1b8f-41bf-ab15-269b399b4147 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.210885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a2c44d-1b8f-41bf-ab15-269b399b4147 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.211179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a2c44d-1b8f-41bf-ab15-269b399b4147 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.260013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:55.261620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f489f2d8fd54dd6870c05b4d4e9b387'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:55.271825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a2c44d-1b8f-41bf-ab15-269b399b4147 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.273264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a2c44d-1b8f-41bf-ab15-269b399b4147 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.282823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 782/1556] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:55.283682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a2c44d-1b8f-41bf-ab15-269b399b4147 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.293282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a2c44d-1b8f-41bf-ab15-269b399b4147 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.298069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec38086-6cbf-49eb-b48f-f860568b3861 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.298789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec38086-6cbf-49eb-b48f-f860568b3861 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.299173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec38086-6cbf-49eb-b48f-f860568b3861 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.299788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec38086-6cbf-49eb-b48f-f860568b3861 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.300075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec38086-6cbf-49eb-b48f-f860568b3861 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.302871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec38086-6cbf-49eb-b48f-f860568b3861 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.327981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f489f2d8fd54dd6870c05b4d4e9b387'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:55.344439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f489f2d8fd54dd6870c05b4d4e9b387'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:55.354010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f489f2d8fd54dd6870c05b4d4e9b387'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:55.355527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f489f2d8fd54dd6870c05b4d4e9b387'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:55.361036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 775/1557] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:55 2022] DELETE /identity/v3/users/2f489f2d8fd54dd6870c05b4d4e9b387 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:55.433925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 783/1558] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:55 2022] POST /identity/v3/users => generated 281 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:55.438247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.439894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:55.476993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:55.509030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fc6f51-42ab-4127-83cd-f4a3ca151586 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.514021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4e77ed-9db5-4000-82b5-293ffe01abc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.514628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e77ed-9db5-4000-82b5-293ffe01abc7 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.514905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e77ed-9db5-4000-82b5-293ffe01abc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.515276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4e77ed-9db5-4000-82b5-293ffe01abc7 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.515954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4e77ed-9db5-4000-82b5-293ffe01abc7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.517583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4e77ed-9db5-4000-82b5-293ffe01abc7 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.532082 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 776/1559] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:55 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:55.532859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec38086-6cbf-49eb-b48f-f860568b3861 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.533538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:55.565685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec38086-6cbf-49eb-b48f-f860568b3861 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.573214 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33353cf-727d-4eff-8e94-1e888049a316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.573974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33353cf-727d-4eff-8e94-1e888049a316 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.574352 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33353cf-727d-4eff-8e94-1e888049a316 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.574724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33353cf-727d-4eff-8e94-1e888049a316 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.575508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33353cf-727d-4eff-8e94-1e888049a316 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.577424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33353cf-727d-4eff-8e94-1e888049a316 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.598015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 784/1560] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:55 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:55.602478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4e77ed-9db5-4000-82b5-293ffe01abc7 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.615223 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4e77ed-9db5-4000-82b5-293ffe01abc7 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.621083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.627377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.627377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.627377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] PATH_INFO: `/v3/projects/28a7bd12f91f4a98b2d601ac8a9519ae/users/bb1ac9433a7c4069818da6c538702df5/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.638940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28a7bd12f91f4a98b2d601ac8a9519ae, user_id=bb1ac9433a7c4069818da6c538702df5, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.641779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.768143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 777/1561] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:55 2022] PUT /identity/v3/projects/28a7bd12f91f4a98b2d601ac8a9519ae/users/bb1ac9433a7c4069818da6c538702df5/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:55.779662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547119d6-a9df-4f4f-80c7-05d05dc04962 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.781178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547119d6-a9df-4f4f-80c7-05d05dc04962 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.781684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547119d6-a9df-4f4f-80c7-05d05dc04962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.783164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.785183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:55.829048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-547119d6-a9df-4f4f-80c7-05d05dc04962 None None] MFA Rules not processed for user `bb1ac9433a7c4069818da6c538702df5`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:55.830321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-547119d6-a9df-4f4f-80c7-05d05dc04962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:55.876141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60001052-8e91-4237-86aa-ad4ecbf31c15 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.885410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.885410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.885410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.885410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] PATH_INFO: `/v3/projects/d6acb1c06c264bb1ab14512f62fcd1fd` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.885760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d6acb1c06c264bb1ab14512f62fcd1fd)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.886852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:55.899850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 785/1562] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:55 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:55.928830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547119d6-a9df-4f4f-80c7-05d05dc04962 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:55.930584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-547119d6-a9df-4f4f-80c7-05d05dc04962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:55.947159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547119d6-a9df-4f4f-80c7-05d05dc04962 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:55.952781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:55.954745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:55.954745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:55.954745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] PATH_INFO: `/v3/projects/265a0e121e7342f7a801a8bf0bf002ba` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:55.955861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=265a0e121e7342f7a801a8bf0bf002ba)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:55.958718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:56.013803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd6acb1c06c264bb1ab14512f62fcd1fd', 'request_id': 'req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.084915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '265a0e121e7342f7a801a8bf0bf002ba', 'request_id': 'req-2c2823d8-266a-4e56-849d-33ec41e6f6ba'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.088601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd6acb1c06c264bb1ab14512f62fcd1fd', 'request_id': 'req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.092289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '265a0e121e7342f7a801a8bf0bf002ba', 'request_id': 'req-2c2823d8-266a-4e56-849d-33ec41e6f6ba'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.095750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd6acb1c06c264bb1ab14512f62fcd1fd', 'request_id': 'req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.099403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '265a0e121e7342f7a801a8bf0bf002ba', 'request_id': 'req-2c2823d8-266a-4e56-849d-33ec41e6f6ba'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.107966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] The token cache is being invalidate because project d6acb1c06c264bb1ab14512f62fcd1fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:56.108259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.108926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.110543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 778/1563] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:55 2022] DELETE /identity/v3/projects/d6acb1c06c264bb1ab14512f62fcd1fd => generated 0 bytes in 328 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:56.111341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:56.113576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:56.171504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ea32f9-7c02-47f3-9f1d-dbb7e70792c3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:56.194743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] The token cache is being invalidate because project 265a0e121e7342f7a801a8bf0bf002ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:56.194743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.194743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.194743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 786/1564] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:55 2022] DELETE /identity/v3/projects/265a0e121e7342f7a801a8bf0bf002ba => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:56.195761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:56.197080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:56.227229 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17e48ff0-8d32-46eb-b452-7a9140b5ca52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:56.227938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e48ff0-8d32-46eb-b452-7a9140b5ca52 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.228228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e48ff0-8d32-46eb-b452-7a9140b5ca52 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.228505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e48ff0-8d32-46eb-b452-7a9140b5ca52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.230638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-17e48ff0-8d32-46eb-b452-7a9140b5ca52 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:56.276029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2823d8-266a-4e56-849d-33ec41e6f6ba admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:56.280896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:56.281529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.281836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.282108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] PATH_INFO: `/v3/users/ba21641b2bab46058a9e22107b5581d3` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.283502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ba21641b2bab46058a9e22107b5581d3)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:56.285697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:56.292008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e48ff0-8d32-46eb-b452-7a9140b5ca52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:56.293484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17e48ff0-8d32-46eb-b452-7a9140b5ca52 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:56.303489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 779/1565] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:56 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:56.304549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca7c7f6-d4e6-4506-b046-d4bc1bf69441 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.304834 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca7c7f6-d4e6-4506-b046-d4bc1bf69441 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.305111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca7c7f6-d4e6-4506-b046-d4bc1bf69441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.326852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-eca7c7f6-d4e6-4506-b046-d4bc1bf69441 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:56.328171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-eca7c7f6-d4e6-4506-b046-d4bc1bf69441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:56.336546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 780/1566] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:56.337961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca7c7f6-d4e6-4506-b046-d4bc1bf69441 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:56.338845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-eca7c7f6-d4e6-4506-b046-d4bc1bf69441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:56.351404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca7c7f6-d4e6-4506-b046-d4bc1bf69441 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:56.356633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:56.361035 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:56.361216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.361216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.361216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] PATH_INFO: `/v3/users/bb1ac9433a7c4069818da6c538702df5` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.361216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb1ac9433a7c4069818da6c538702df5)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:56.361216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:56.435607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:56.533358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ba21641b2bab46058a9e22107b5581d3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.534114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ba21641b2bab46058a9e22107b5581d3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.549076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ba21641b2bab46058a9e22107b5581d3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.558001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ba21641b2bab46058a9e22107b5581d3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.558822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b24030a6-d57d-49d6-a3f6-7a495c502205 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ba21641b2bab46058a9e22107b5581d3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.587438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb1ac9433a7c4069818da6c538702df5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.653614 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 787/1567] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:56 2022] DELETE /identity/v3/users/ba21641b2bab46058a9e22107b5581d3 => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:56.655818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5af21d-86fd-4a8a-9ffe-f3fbd73cc8d5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.656471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5af21d-86fd-4a8a-9ffe-f3fbd73cc8d5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.656817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5af21d-86fd-4a8a-9ffe-f3fbd73cc8d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.684438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-cc5af21d-86fd-4a8a-9ffe-f3fbd73cc8d5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:56.685689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cc5af21d-86fd-4a8a-9ffe-f3fbd73cc8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:56.698002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb1ac9433a7c4069818da6c538702df5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.713809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb1ac9433a7c4069818da6c538702df5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.722179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb1ac9433a7c4069818da6c538702df5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.722931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-6257cd49-9855-41fa-8cfb-0ce679ec60d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb1ac9433a7c4069818da6c538702df5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:56.729801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 781/1568] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:02:56 2022] DELETE /identity/v3/users/bb1ac9433a7c4069818da6c538702df5 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:56.734308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c51bdf-1461-41b0-ab60-747073d13805 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.734701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c51bdf-1461-41b0-ab60-747073d13805 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.735019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c51bdf-1461-41b0-ab60-747073d13805 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.750892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 788/1569] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:56.755744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e3c51bdf-1461-41b0-ab60-747073d13805 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:56.756969 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e3c51bdf-1461-41b0-ab60-747073d13805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:56.761088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5af21d-86fd-4a8a-9ffe-f3fbd73cc8d5 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:56.770138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5af21d-86fd-4a8a-9ffe-f3fbd73cc8d5 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:56.820709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 782/1570] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:56.828370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:56.828370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.828597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.828965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.829399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:56.829709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:56.831630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:56.836784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c51bdf-1461-41b0-ab60-747073d13805 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:56.837520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e3c51bdf-1461-41b0-ab60-747073d13805 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:56.851105 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 789/1571] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:02:56.851871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:56.867561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ada4bc0-e3b9-470a-a452-32395f4632e9 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:56.876899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:56.878157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.879095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.879661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.881203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:56.884824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:56.910026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c51bdf-1461-41b0-ab60-747073d13805 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:56.915246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfcd9ed-b46d-4863-b093-eaa10254695f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:56.915924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfcd9ed-b46d-4863-b093-eaa10254695f service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:56.916209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfcd9ed-b46d-4863-b093-eaa10254695f service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:56.916484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfcd9ed-b46d-4863-b093-eaa10254695f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:56.928898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfcd9ed-b46d-4863-b093-eaa10254695f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:56.932522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:56.935861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfcd9ed-b46d-4863-b093-eaa10254695f service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:56.944161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 783/1572] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:56.985644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:56.992485 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 790/1573] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:56 2022] POST /identity/v3/projects => generated 407 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:02:56.996259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfcd9ed-b46d-4863-b093-eaa10254695f service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.011842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfcd9ed-b46d-4863-b093-eaa10254695f service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.096113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9d5fcb-ac1d-498b-acf7-78050ee5f0db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.096801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9d5fcb-ac1d-498b-acf7-78050ee5f0db admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.097164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9d5fcb-ac1d-498b-acf7-78050ee5f0db admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.097505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9d5fcb-ac1d-498b-acf7-78050ee5f0db admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.098821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9d5fcb-ac1d-498b-acf7-78050ee5f0db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.100903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9d5fcb-ac1d-498b-acf7-78050ee5f0db admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.135140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.137086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:57.150671 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f743096d-4042-47b0-b158-9f5ea9577766 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.155634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.156363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.156712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.157078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] PATH_INFO: `/v3/projects/6876cbdeea004273ba55aa6b16ed73a5` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.158264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] RBAC: Authorizing `identity:delete_project(project_id=6876cbdeea004273ba55aa6b16ed73a5)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.160283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.200095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 784/1574] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:56 2022] POST /identity/v3/users => generated 304 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:57.204017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9d5fcb-ac1d-498b-acf7-78050ee5f0db admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.228584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9d5fcb-ac1d-498b-acf7-78050ee5f0db admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.233865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0343c866-79d5-4da5-af19-c2f8b660ec6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.234704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343c866-79d5-4da5-af19-c2f8b660ec6c admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.234921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343c866-79d5-4da5-af19-c2f8b660ec6c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.235315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0343c866-79d5-4da5-af19-c2f8b660ec6c admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.236430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0343c866-79d5-4da5-af19-c2f8b660ec6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.238267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0343c866-79d5-4da5-af19-c2f8b660ec6c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.261071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 785/1575] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:57.262327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0343c866-79d5-4da5-af19-c2f8b660ec6c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.311577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0343c866-79d5-4da5-af19-c2f8b660ec6c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.324065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6876cbdeea004273ba55aa6b16ed73a5', 'request_id': 'req-53c58076-a1ba-4c2d-bc03-c2564e0658df'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.334713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6876cbdeea004273ba55aa6b16ed73a5', 'request_id': 'req-53c58076-a1ba-4c2d-bc03-c2564e0658df'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.354501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6876cbdeea004273ba55aa6b16ed73a5', 'request_id': 'req-53c58076-a1ba-4c2d-bc03-c2564e0658df'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.411342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd9fd2d-8d4a-4570-a3ea-63199cb5dc36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.415485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd9fd2d-8d4a-4570-a3ea-63199cb5dc36 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.417793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd9fd2d-8d4a-4570-a3ea-63199cb5dc36 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.417793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd9fd2d-8d4a-4570-a3ea-63199cb5dc36 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.417793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd9fd2d-8d4a-4570-a3ea-63199cb5dc36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.419167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd9fd2d-8d4a-4570-a3ea-63199cb5dc36 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.436628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 786/1576] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:57 2022] GET /identity/v3/roles => generated 1420 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:02:57.438216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd9fd2d-8d4a-4570-a3ea-63199cb5dc36 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.445801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] The token cache is being invalidate because project 6876cbdeea004273ba55aa6b16ed73a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:57.446166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.447908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-53c58076-a1ba-4c2d-bc03-c2564e0658df admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.449821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 791/1577] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:57 2022] DELETE /identity/v3/projects/6876cbdeea004273ba55aa6b16ed73a5 => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:02:57.458680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b999afa8-3fc1-46f9-8fac-47846ded6f21 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.458979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b999afa8-3fc1-46f9-8fac-47846ded6f21 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.459350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b999afa8-3fc1-46f9-8fac-47846ded6f21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.463856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd9fd2d-8d4a-4570-a3ea-63199cb5dc36 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.470328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ea69d5-ab21-498b-b2e7-7e7beadc53ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.471028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ea69d5-ab21-498b-b2e7-7e7beadc53ed admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.471314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ea69d5-ab21-498b-b2e7-7e7beadc53ed admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.471594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ea69d5-ab21-498b-b2e7-7e7beadc53ed admin admin] PATH_INFO: `/v3/projects/3aa3c9449d0346cc8a0917a487274123/users/3bb0c59ee869439798d6bbcfac57706d/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.483406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b999afa8-3fc1-46f9-8fac-47846ded6f21 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:57.484681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b999afa8-3fc1-46f9-8fac-47846ded6f21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:57.486639 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ea69d5-ab21-498b-b2e7-7e7beadc53ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=3aa3c9449d0346cc8a0917a487274123, user_id=3bb0c59ee869439798d6bbcfac57706d, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.489860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ea69d5-ab21-498b-b2e7-7e7beadc53ed admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.500021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 792/1578] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:57.500790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d28bc-39c2-41f7-89f0-c286bcbaafeb None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.501066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d28bc-39c2-41f7-89f0-c286bcbaafeb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.501334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e0d28bc-39c2-41f7-89f0-c286bcbaafeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.521508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5e0d28bc-39c2-41f7-89f0-c286bcbaafeb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:57.522690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5e0d28bc-39c2-41f7-89f0-c286bcbaafeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:57.535933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 793/1579] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:02:57.535933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0d28bc-39c2-41f7-89f0-c286bcbaafeb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.535933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5e0d28bc-39c2-41f7-89f0-c286bcbaafeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:57.548246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e0d28bc-39c2-41f7-89f0-c286bcbaafeb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.553363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.553993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.554273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.554547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] PATH_INFO: `/v3/projects/28a7bd12f91f4a98b2d601ac8a9519ae` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.555703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] RBAC: Authorizing `identity:delete_project(project_id=28a7bd12f91f4a98b2d601ac8a9519ae)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.557611 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.591025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 787/1580] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:57 2022] PUT /identity/v3/projects/3aa3c9449d0346cc8a0917a487274123/users/3bb0c59ee869439798d6bbcfac57706d/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:57.592046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ea69d5-ab21-498b-b2e7-7e7beadc53ed admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.596984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=6,keystone:SELECT=92,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:57.652975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ea69d5-ab21-498b-b2e7-7e7beadc53ed admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.657797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af307361-ca44-41db-955c-b7c0f0e076c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.658405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af307361-ca44-41db-955c-b7c0f0e076c3 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.658684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af307361-ca44-41db-955c-b7c0f0e076c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.658975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af307361-ca44-41db-955c-b7c0f0e076c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.659640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af307361-ca44-41db-955c-b7c0f0e076c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:57.659941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af307361-ca44-41db-955c-b7c0f0e076c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.661546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af307361-ca44-41db-955c-b7c0f0e076c3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.673524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 788/1581] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:57.674711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851a479-158b-4cf2-ac2f-516e7c5c0ec5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.675026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851a479-158b-4cf2-ac2f-516e7c5c0ec5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.675562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851a479-158b-4cf2-ac2f-516e7c5c0ec5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.696245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28a7bd12f91f4a98b2d601ac8a9519ae', 'request_id': 'req-da140a75-f8f0-4ba4-98f2-4d91a24b849c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.703064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28a7bd12f91f4a98b2d601ac8a9519ae', 'request_id': 'req-da140a75-f8f0-4ba4-98f2-4d91a24b849c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.707169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-4851a479-158b-4cf2-ac2f-516e7c5c0ec5 None None] MFA Rules not processed for user `3bb0c59ee869439798d6bbcfac57706d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:57.708500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4851a479-158b-4cf2-ac2f-516e7c5c0ec5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:57.708847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28a7bd12f91f4a98b2d601ac8a9519ae', 'request_id': 'req-da140a75-f8f0-4ba4-98f2-4d91a24b849c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.773245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 789/1582] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:57 2022] POST /identity/v3/auth/tokens => generated 3284 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:57.774157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4851a479-158b-4cf2-ac2f-516e7c5c0ec5 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.786690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] The token cache is being invalidate because project 28a7bd12f91f4a98b2d601ac8a9519ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:02:57.787156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.787996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:02:57.790204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 794/1583] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:02:57 2022] DELETE /identity/v3/projects/28a7bd12f91f4a98b2d601ac8a9519ae => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:57.795690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.797696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:57.855144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4851a479-158b-4cf2-ac2f-516e7c5c0ec5 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.860470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2642a8-e34b-430f-bca7-bd8bd7c9e2ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.861459 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2642a8-e34b-430f-bca7-bd8bd7c9e2ab admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.861877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2642a8-e34b-430f-bca7-bd8bd7c9e2ab admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.862287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2642a8-e34b-430f-bca7-bd8bd7c9e2ab admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.863117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2642a8-e34b-430f-bca7-bd8bd7c9e2ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.865728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2642a8-e34b-430f-bca7-bd8bd7c9e2ab admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.867669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da140a75-f8f0-4ba4-98f2-4d91a24b849c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:57.872469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:57.873072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.873351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.873677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.874321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:57.876342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:57.910889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:57.969101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 790/1584] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:57 2022] POST /identity/v3/projects => generated 387 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:57.970346 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ec36ca-8c7a-49ce-9d7c-539e1bc83f8a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:57.970628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ec36ca-8c7a-49ce-9d7c-539e1bc83f8a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:57.970901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6ec36ca-8c7a-49ce-9d7c-539e1bc83f8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:57.975461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 795/1585] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:57 2022] POST /identity/v3/projects => generated 409 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:57.976320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:57.977974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:57.993658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b6ec36ca-8c7a-49ce-9d7c-539e1bc83f8a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:57.994855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b6ec36ca-8c7a-49ce-9d7c-539e1bc83f8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:58.042394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4727d15-6d4d-4ea4-91ba-accce5664b64 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.048472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b12e0a-c00f-419c-a291-c51f7db560cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.048472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b12e0a-c00f-419c-a291-c51f7db560cc admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.048472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b12e0a-c00f-419c-a291-c51f7db560cc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.048472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b12e0a-c00f-419c-a291-c51f7db560cc admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.055661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b12e0a-c00f-419c-a291-c51f7db560cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.061183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b12e0a-c00f-419c-a291-c51f7db560cc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.069684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 791/1586] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:58.069684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ec36ca-8c7a-49ce-9d7c-539e1bc83f8a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:58.088089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6ec36ca-8c7a-49ce-9d7c-539e1bc83f8a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.096718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26ecf3-9c65-4a4e-a477-4041c791cf34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.097451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26ecf3-9c65-4a4e-a477-4041c791cf34 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.097734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26ecf3-9c65-4a4e-a477-4041c791cf34 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.098005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a26ecf3-9c65-4a4e-a477-4041c791cf34 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.099195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26ecf3-9c65-4a4e-a477-4041c791cf34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.102030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a26ecf3-9c65-4a4e-a477-4041c791cf34 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.488367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 796/1588] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:57 2022] POST /identity/v3/users => generated 294 bytes in 510 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:58.488916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 792/1588] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:58 2022] POST /identity/v3/users => generated 311 bytes in 421 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:58.488916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26ecf3-9c65-4a4e-a477-4041c791cf34 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:58.490207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d11c4d-7fd0-40c3-8bef-7eaea98cd11b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.490550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d11c4d-7fd0-40c3-8bef-7eaea98cd11b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.491283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d11c4d-7fd0-40c3-8bef-7eaea98cd11b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.512167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a26ecf3-9c65-4a4e-a477-4041c791cf34 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.525423 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e5cbd2-4941-48b6-89aa-5d228fb68a30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.527138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e5cbd2-4941-48b6-89aa-5d228fb68a30 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.530117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e5cbd2-4941-48b6-89aa-5d228fb68a30 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.530117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e5cbd2-4941-48b6-89aa-5d228fb68a30 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.531991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e5cbd2-4941-48b6-89aa-5d228fb68a30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.537752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e5cbd2-4941-48b6-89aa-5d228fb68a30 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.540006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-56d11c4d-7fd0-40c3-8bef-7eaea98cd11b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:58.541279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-56d11c4d-7fd0-40c3-8bef-7eaea98cd11b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:58.567340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 797/1589] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:58.569478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d11c4d-7fd0-40c3-8bef-7eaea98cd11b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:58.574579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 793/1590] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:58.575779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e5cbd2-4941-48b6-89aa-5d228fb68a30 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:58.590207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e5cbd2-4941-48b6-89aa-5d228fb68a30 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.597320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.598760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.600067 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.600431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.601753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:02:58.602232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.604066 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.605262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d11c4d-7fd0-40c3-8bef-7eaea98cd11b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.626882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03bf72e-b86a-4be6-80e1-b7de15a085ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.627535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03bf72e-b86a-4be6-80e1-b7de15a085ba admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.627819 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03bf72e-b86a-4be6-80e1-b7de15a085ba admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.628093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03bf72e-b86a-4be6-80e1-b7de15a085ba admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.628747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03bf72e-b86a-4be6-80e1-b7de15a085ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.631534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 794/1591] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:02:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:02:58.634919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:58.635080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03bf72e-b86a-4be6-80e1-b7de15a085ba admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.643289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120546b7-683b-4ae4-bf24-6400239d3008 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.648754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc845a1-e5df-484f-839e-9003d23e8c66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.649531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc845a1-e5df-484f-839e-9003d23e8c66 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.649907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc845a1-e5df-484f-839e-9003d23e8c66 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.650239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc845a1-e5df-484f-839e-9003d23e8c66 admin admin] PATH_INFO: `/v3/projects/c63a114f1bca481984136b9ba64c58a0/users/6709098dab7d4dbf9da2a8f0bde7ff06/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.661378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 798/1592] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:58.661378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03bf72e-b86a-4be6-80e1-b7de15a085ba admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:58.672905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03bf72e-b86a-4be6-80e1-b7de15a085ba admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.675931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc845a1-e5df-484f-839e-9003d23e8c66 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c63a114f1bca481984136b9ba64c58a0, user_id=6709098dab7d4dbf9da2a8f0bde7ff06, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.677368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc845a1-e5df-484f-839e-9003d23e8c66 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.679522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d3f406-894f-4387-8a98-351ee546098c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.680205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d3f406-894f-4387-8a98-351ee546098c admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.680810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d3f406-894f-4387-8a98-351ee546098c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.687396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d3f406-894f-4387-8a98-351ee546098c admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.688116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d3f406-894f-4387-8a98-351ee546098c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.690183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d3f406-894f-4387-8a98-351ee546098c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.786563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 795/1593] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:58 2022] PUT /identity/v3/projects/c63a114f1bca481984136b9ba64c58a0/users/6709098dab7d4dbf9da2a8f0bde7ff06/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:58.787236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc845a1-e5df-484f-839e-9003d23e8c66 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:58.805182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 799/1594] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:58 2022] POST /identity/v3/projects => generated 385 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:58.805182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d3f406-894f-4387-8a98-351ee546098c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:58.951666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc845a1-e5df-484f-839e-9003d23e8c66 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.968917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d3f406-894f-4387-8a98-351ee546098c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:58.972681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01833862-d4d4-4771-aed2-edb8131d8fbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.973342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01833862-d4d4-4771-aed2-edb8131d8fbd admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.973621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01833862-d4d4-4771-aed2-edb8131d8fbd admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.973895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01833862-d4d4-4771-aed2-edb8131d8fbd admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.974556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01833862-d4d4-4771-aed2-edb8131d8fbd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.977401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01833862-d4d4-4771-aed2-edb8131d8fbd admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.981877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3482c1e-15a8-49bf-8432-35d555957fd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:58.982944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3482c1e-15a8-49bf-8432-35d555957fd5 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:58.983380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3482c1e-15a8-49bf-8432-35d555957fd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:58.983848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3482c1e-15a8-49bf-8432-35d555957fd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:58.984652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3482c1e-15a8-49bf-8432-35d555957fd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:58.986492 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3482c1e-15a8-49bf-8432-35d555957fd5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:58.997580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 796/1595] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:58.997580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01833862-d4d4-4771-aed2-edb8131d8fbd admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.005818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 800/1596] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:58 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:59.008015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3482c1e-15a8-49bf-8432-35d555957fd5 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.012962 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01833862-d4d4-4771-aed2-edb8131d8fbd admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.018620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de5cc58-8103-4269-a653-8bae45ec1e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.019350 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5cc58-8103-4269-a653-8bae45ec1e23 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.019633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5cc58-8103-4269-a653-8bae45ec1e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.020578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6de5cc58-8103-4269-a653-8bae45ec1e23 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.021340 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de5cc58-8103-4269-a653-8bae45ec1e23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.023644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6de5cc58-8103-4269-a653-8bae45ec1e23 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.036682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3482c1e-15a8-49bf-8432-35d555957fd5 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.050938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:59.053408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f866fff5-e5cb-4b54-a7a0-bb359ea1fdd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.054852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f866fff5-e5cb-4b54-a7a0-bb359ea1fdd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.055181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f866fff5-e5cb-4b54-a7a0-bb359ea1fdd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.055460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f866fff5-e5cb-4b54-a7a0-bb359ea1fdd9 admin admin] PATH_INFO: `/v3/projects/78776822837f4137843d6ebed5f6d002/users/81c7c3418b0b4acebf2744698c31db9d/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.075686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f866fff5-e5cb-4b54-a7a0-bb359ea1fdd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78776822837f4137843d6ebed5f6d002, user_id=81c7c3418b0b4acebf2744698c31db9d, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.078488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f866fff5-e5cb-4b54-a7a0-bb359ea1fdd9 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.123497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 797/1597] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:58 2022] POST /identity/v3/users => generated 293 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:59.124369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de5cc58-8103-4269-a653-8bae45ec1e23 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.136600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6de5cc58-8103-4269-a653-8bae45ec1e23 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.143396 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992ece17-1782-4c4c-b052-17857c97529c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.146666 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 801/1598] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:59 2022] PUT /identity/v3/projects/78776822837f4137843d6ebed5f6d002/users/81c7c3418b0b4acebf2744698c31db9d/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:59.147331 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f866fff5-e5cb-4b54-a7a0-bb359ea1fdd9 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.154662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ece17-1782-4c4c-b052-17857c97529c admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.155063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ece17-1782-4c4c-b052-17857c97529c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.155494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992ece17-1782-4c4c-b052-17857c97529c admin admin] PATH_INFO: `/v3/projects/c63a114f1bca481984136b9ba64c58a0/users/6709098dab7d4dbf9da2a8f0bde7ff06/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.163076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992ece17-1782-4c4c-b052-17857c97529c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c63a114f1bca481984136b9ba64c58a0, user_id=6709098dab7d4dbf9da2a8f0bde7ff06, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.163076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992ece17-1782-4c4c-b052-17857c97529c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.186785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 798/1599] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:59 2022] PUT /identity/v3/projects/c63a114f1bca481984136b9ba64c58a0/users/6709098dab7d4dbf9da2a8f0bde7ff06/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:59.186785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88e893e-d34d-4322-accd-2dc9cece9a1c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.186785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88e893e-d34d-4322-accd-2dc9cece9a1c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.186785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88e893e-d34d-4322-accd-2dc9cece9a1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.221722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c88e893e-d34d-4322-accd-2dc9cece9a1c None None] MFA Rules not processed for user `81c7c3418b0b4acebf2744698c31db9d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:59.222987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c88e893e-d34d-4322-accd-2dc9cece9a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:59.233872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f866fff5-e5cb-4b54-a7a0-bb359ea1fdd9 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637fcb6b-248f-4c75-a86a-7ff45c5432a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.433562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:59.433822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637fcb6b-248f-4c75-a86a-7ff45c5432a3 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.434102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637fcb6b-248f-4c75-a86a-7ff45c5432a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.434373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637fcb6b-248f-4c75-a86a-7ff45c5432a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.435044 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637fcb6b-248f-4c75-a86a-7ff45c5432a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.436730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637fcb6b-248f-4c75-a86a-7ff45c5432a3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.450611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 799/1600] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:59 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 268 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:59.452786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 802/1601] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:59.452953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6033087-7989-4ed4-8432-049bcc672fd7 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.452953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6033087-7989-4ed4-8432-049bcc672fd7 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.452953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6033087-7989-4ed4-8432-049bcc672fd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.453235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637fcb6b-248f-4c75-a86a-7ff45c5432a3 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.462789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637fcb6b-248f-4c75-a86a-7ff45c5432a3 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.467488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc88c928-357e-4be7-9646-87c7dc9760d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.468098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88c928-357e-4be7-9646-87c7dc9760d4 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.468441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88c928-357e-4be7-9646-87c7dc9760d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.468644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc88c928-357e-4be7-9646-87c7dc9760d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.469280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc88c928-357e-4be7-9646-87c7dc9760d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.470861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc88c928-357e-4be7-9646-87c7dc9760d4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.478539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c6033087-7989-4ed4-8432-049bcc672fd7 None None] MFA Rules not processed for user `6709098dab7d4dbf9da2a8f0bde7ff06`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:59.484055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c6033087-7989-4ed4-8432-049bcc672fd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:59.484737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 803/1602] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:02:59.489426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc88c928-357e-4be7-9646-87c7dc9760d4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.499075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc88c928-357e-4be7-9646-87c7dc9760d4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.505677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.506282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.506561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.506881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.508496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.509531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.554003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 800/1603] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:59 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:59.554823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6033087-7989-4ed4-8432-049bcc672fd7 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.565240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6033087-7989-4ed4-8432-049bcc672fd7 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.569866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26c283-39fd-48c9-af8a-e22eb6357068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.570480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c283-39fd-48c9-af8a-e22eb6357068 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.570760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c283-39fd-48c9-af8a-e22eb6357068 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.571058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c283-39fd-48c9-af8a-e22eb6357068 admin admin] PATH_INFO: `/v3/projects/90c65ab417f14b209fe154f3e9e74516/users/0116b22b9dd143d3b74b9b56989cd7fa/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.585298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26c283-39fd-48c9-af8a-e22eb6357068 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90c65ab417f14b209fe154f3e9e74516, user_id=0116b22b9dd143d3b74b9b56989cd7fa, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.588004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26c283-39fd-48c9-af8a-e22eb6357068 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.596517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 804/1604] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:02:59 2022] POST /identity/v3/projects => generated 387 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:59.598693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.599579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:59.655000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246ae6e4-a259-463a-971c-98fbc244b08c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.658879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380478d2-b7ff-4aca-9029-6b6976e6d0d7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.659570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380478d2-b7ff-4aca-9029-6b6976e6d0d7 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.659851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380478d2-b7ff-4aca-9029-6b6976e6d0d7 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.660129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380478d2-b7ff-4aca-9029-6b6976e6d0d7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.682219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 801/1605] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:59 2022] PUT /identity/v3/projects/90c65ab417f14b209fe154f3e9e74516/users/0116b22b9dd143d3b74b9b56989cd7fa/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:02:59.685071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26c283-39fd-48c9-af8a-e22eb6357068 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.725344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380478d2-b7ff-4aca-9029-6b6976e6d0d7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.726765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380478d2-b7ff-4aca-9029-6b6976e6d0d7 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.737167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 805/1606] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:59 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:02:59.738261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0c9622-6747-4d1e-a3ae-0ffb9f70bd78 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.738550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0c9622-6747-4d1e-a3ae-0ffb9f70bd78 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.739433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0c9622-6747-4d1e-a3ae-0ffb9f70bd78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.769155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26c283-39fd-48c9-af8a-e22eb6357068 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.772725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5d0c9622-6747-4d1e-a3ae-0ffb9f70bd78 None None] MFA Rules not processed for user `0116b22b9dd143d3b74b9b56989cd7fa`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:02:59.774300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5d0c9622-6747-4d1e-a3ae-0ffb9f70bd78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:02:59.774477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b51b9-4543-4b0e-9f77-8e496f3a9c8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.775300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b51b9-4543-4b0e-9f77-8e496f3a9c8b admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.775582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b51b9-4543-4b0e-9f77-8e496f3a9c8b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.775852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b51b9-4543-4b0e-9f77-8e496f3a9c8b admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.776502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b51b9-4543-4b0e-9f77-8e496f3a9c8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.779735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b51b9-4543-4b0e-9f77-8e496f3a9c8b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.863562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 806/1607] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:02:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:02:59.876288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 802/1608] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:02:59 2022] POST /identity/v3/users => generated 300 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:02:59.878310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0c9622-6747-4d1e-a3ae-0ffb9f70bd78 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.881444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:59.886774 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b51b9-4543-4b0e-9f77-8e496f3a9c8b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.903551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b51b9-4543-4b0e-9f77-8e496f3a9c8b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.912521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf191deb-b9d2-4db7-8318-f742d0ead75b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.913592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf191deb-b9d2-4db7-8318-f742d0ead75b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.913883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf191deb-b9d2-4db7-8318-f742d0ead75b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.914162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf191deb-b9d2-4db7-8318-f742d0ead75b admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.914989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf191deb-b9d2-4db7-8318-f742d0ead75b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.917218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf191deb-b9d2-4db7-8318-f742d0ead75b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.933083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 803/1609] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:02:59 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:02:59.933997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf191deb-b9d2-4db7-8318-f742d0ead75b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:02:59.945196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf191deb-b9d2-4db7-8318-f742d0ead75b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.947866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0c9622-6747-4d1e-a3ae-0ffb9f70bd78 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:02:59.950203 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4778ea34-4224-44b7-9aa9-9cb3f8f6ca54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.950857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778ea34-4224-44b7-9aa9-9cb3f8f6ca54 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.951171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778ea34-4224-44b7-9aa9-9cb3f8f6ca54 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.951449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4778ea34-4224-44b7-9aa9-9cb3f8f6ca54 admin admin] PATH_INFO: `/v3/projects/01fc552a4d824ab38277754c183e8ae4/users/b20aed9d28c8439b80fa58e698b1ea01/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.952220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e0a31a-176d-4c44-a181-ff07a4fb36e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:02:59.952818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e0a31a-176d-4c44-a181-ff07a4fb36e0 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:02:59.953155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e0a31a-176d-4c44-a181-ff07a4fb36e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:02:59.953490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9e0a31a-176d-4c44-a181-ff07a4fb36e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:02:59.962850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:02:59.964375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e0a31a-176d-4c44-a181-ff07a4fb36e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.965800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9e0a31a-176d-4c44-a181-ff07a4fb36e0 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.968066 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4778ea34-4224-44b7-9aa9-9cb3f8f6ca54 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01fc552a4d824ab38277754c183e8ae4, user_id=b20aed9d28c8439b80fa58e698b1ea01, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:02:59.970785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4778ea34-4224-44b7-9aa9-9cb3f8f6ca54 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:02:59.975743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 807/1610] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:02:59 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:00.048150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 804/1611] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:02:59 2022] PUT /identity/v3/projects/01fc552a4d824ab38277754c183e8ae4/users/b20aed9d28c8439b80fa58e698b1ea01/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:00.052339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e0a31a-176d-4c44-a181-ff07a4fb36e0 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:00.063018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9e0a31a-176d-4c44-a181-ff07a4fb36e0 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:00.121549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab56b63e-4e68-4b05-9f77-fa91e8d517ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:00.122071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56b63e-4e68-4b05-9f77-fa91e8d517ed admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:00.122348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56b63e-4e68-4b05-9f77-fa91e8d517ed admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:00.122615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56b63e-4e68-4b05-9f77-fa91e8d517ed admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:00.123277 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab56b63e-4e68-4b05-9f77-fa91e8d517ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:00.124854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab56b63e-4e68-4b05-9f77-fa91e8d517ed admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:00.134853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 808/1612] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:00 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:00.139236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4778ea34-4224-44b7-9aa9-9cb3f8f6ca54 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:00.150597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4778ea34-4224-44b7-9aa9-9cb3f8f6ca54 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:00.155450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07924f5d-86bf-44a7-9abf-62cacf2d7ea2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:00.156179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07924f5d-86bf-44a7-9abf-62cacf2d7ea2 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:00.156541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07924f5d-86bf-44a7-9abf-62cacf2d7ea2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:00.156884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07924f5d-86bf-44a7-9abf-62cacf2d7ea2 admin admin] PATH_INFO: `/v3/projects/01fc552a4d824ab38277754c183e8ae4/users/b20aed9d28c8439b80fa58e698b1ea01/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:00.158884 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07924f5d-86bf-44a7-9abf-62cacf2d7ea2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=01fc552a4d824ab38277754c183e8ae4, user_id=b20aed9d28c8439b80fa58e698b1ea01, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:00.162213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07924f5d-86bf-44a7-9abf-62cacf2d7ea2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:00.172372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 805/1613] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:00 2022] PUT /identity/v3/projects/01fc552a4d824ab38277754c183e8ae4/users/b20aed9d28c8439b80fa58e698b1ea01/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:00.188294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f360d2-a472-4fa2-ad83-78800ee1ae61 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:00.188294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f360d2-a472-4fa2-ad83-78800ee1ae61 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:00.188294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f360d2-a472-4fa2-ad83-78800ee1ae61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:00.224724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f3f360d2-a472-4fa2-ad83-78800ee1ae61 None None] MFA Rules not processed for user `b20aed9d28c8439b80fa58e698b1ea01`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:00.225957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f3f360d2-a472-4fa2-ad83-78800ee1ae61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:00.237712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:00.311051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 809/1614] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:00 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:00.328616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:00.598241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07924f5d-86bf-44a7-9abf-62cacf2d7ea2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:00.654219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07924f5d-86bf-44a7-9abf-62cacf2d7ea2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:00.657805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:00.658402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:00.658682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:00.658976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:00.660725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:00.718113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:00.724893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:00.726432 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:00.736361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 806/1615] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:00 2022] GET /identity/v3/auth/tokens => generated 3284 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:00.789246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f360d2-a472-4fa2-ad83-78800ee1ae61 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:00.790784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f3f360d2-a472-4fa2-ad83-78800ee1ae61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:00.806898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f360d2-a472-4fa2-ad83-78800ee1ae61 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:00.811306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a32911-f56c-463d-b1f6-422212b0351e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:00.811994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a32911-f56c-463d-b1f6-422212b0351e service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:00.812339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a32911-f56c-463d-b1f6-422212b0351e service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:00.812682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a32911-f56c-463d-b1f6-422212b0351e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:00.823086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a32911-f56c-463d-b1f6-422212b0351e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:00.824593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a32911-f56c-463d-b1f6-422212b0351e service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:00.834771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 810/1616] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:00 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:00.866369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:01.015715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:01.079355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-480ea1c5-6809-4ed8-887e-5c44dc5cf528 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:01.084016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:01.084644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:01.084930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:01.085201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] PATH_INFO: `/v3/users/81c7c3418b0b4acebf2744698c31db9d` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:01.086335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] RBAC: Authorizing `identity:delete_user(user_id=81c7c3418b0b4acebf2744698c31db9d)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:01.088311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:01.284899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '81c7c3418b0b4acebf2744698c31db9d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:01.293818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:01.361987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '81c7c3418b0b4acebf2744698c31db9d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:01.382389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '81c7c3418b0b4acebf2744698c31db9d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:01.393376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '81c7c3418b0b4acebf2744698c31db9d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:01.398659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '81c7c3418b0b4acebf2744698c31db9d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:01.404250 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 807/1617] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:01 2022] DELETE /identity/v3/users/81c7c3418b0b4acebf2744698c31db9d => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:01.408565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba038d-c706-4f84-befd-6f7e30941b3d None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:01.408917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba038d-c706-4f84-befd-6f7e30941b3d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:01.409218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ba038d-c706-4f84-befd-6f7e30941b3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:01.430672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f8ba038d-c706-4f84-befd-6f7e30941b3d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:01.432128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f8ba038d-c706-4f84-befd-6f7e30941b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:01.495537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 811/1618] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:01.505853 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:01.507439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:01.567094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87aef44c-3815-44b4-b67f-5308184cfe5a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:01.571184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccab183-86f1-40e3-b683-0bb898b2fe63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:01.571814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccab183-86f1-40e3-b683-0bb898b2fe63 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:01.572088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccab183-86f1-40e3-b683-0bb898b2fe63 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:01.572355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccab183-86f1-40e3-b683-0bb898b2fe63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:01.581164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccab183-86f1-40e3-b683-0bb898b2fe63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:01.583271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccab183-86f1-40e3-b683-0bb898b2fe63 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:01.592933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 808/1619] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:02.041958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ba038d-c706-4f84-befd-6f7e30941b3d None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:02.043497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f8ba038d-c706-4f84-befd-6f7e30941b3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:02.056966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ba038d-c706-4f84-befd-6f7e30941b3d None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:02.062378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:02.063050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:02.063334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:02.063651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] PATH_INFO: `/v3/projects/78776822837f4137843d6ebed5f6d002` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:02.064959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=78776822837f4137843d6ebed5f6d002)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:02.067378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:02.196860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78776822837f4137843d6ebed5f6d002', 'request_id': 'req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.203439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78776822837f4137843d6ebed5f6d002', 'request_id': 'req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.216528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78776822837f4137843d6ebed5f6d002', 'request_id': 'req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.281728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] The token cache is being invalidate because project 78776822837f4137843d6ebed5f6d002 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:02.281962 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.282637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.284669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 812/1620] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:02 2022] DELETE /identity/v3/projects/78776822837f4137843d6ebed5f6d002 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:02.289046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccab183-86f1-40e3-b683-0bb898b2fe63 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:02.298891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccab183-86f1-40e3-b683-0bb898b2fe63 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:02.356806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:02.357444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:02.357725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:02.357999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] PATH_INFO: `/v3/users/b20aed9d28c8439b80fa58e698b1ea01` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:02.359155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] RBAC: Authorizing `identity:delete_user(user_id=b20aed9d28c8439b80fa58e698b1ea01)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:02.361083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:02.570240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:02.573020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b20aed9d28c8439b80fa58e698b1ea01'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.654783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b20aed9d28c8439b80fa58e698b1ea01'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.670490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b20aed9d28c8439b80fa58e698b1ea01'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.678943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b20aed9d28c8439b80fa58e698b1ea01'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.679776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d84096e9-8d4e-4181-8708-65d57670908c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b20aed9d28c8439b80fa58e698b1ea01'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:02.687637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 809/1621] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:02 2022] DELETE /identity/v3/users/b20aed9d28c8439b80fa58e698b1ea01 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:03.152923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:03.160176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:03.185425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fdadf4-71ad-42e4-9e9a-7019bd05daf5 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:03.199295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:03.271559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:03.279531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:03.279531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:03.279531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] PATH_INFO: `/v3/projects/01fc552a4d824ab38277754c183e8ae4` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:03.279531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] RBAC: Authorizing `identity:delete_project(project_id=01fc552a4d824ab38277754c183e8ae4)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:03.279531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:03.403248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '01fc552a4d824ab38277754c183e8ae4', 'request_id': 'req-68b5e403-35a0-449c-93bd-7c90b6724030'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:03.442982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '01fc552a4d824ab38277754c183e8ae4', 'request_id': 'req-68b5e403-35a0-449c-93bd-7c90b6724030'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:03.442982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '01fc552a4d824ab38277754c183e8ae4', 'request_id': 'req-68b5e403-35a0-449c-93bd-7c90b6724030'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:03.528098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] The token cache is being invalidate because project 01fc552a4d824ab38277754c183e8ae4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:03.528413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:03.529032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-68b5e403-35a0-449c-93bd-7c90b6724030 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:03.530542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 813/1622] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:03 2022] DELETE /identity/v3/projects/01fc552a4d824ab38277754c183e8ae4 => generated 0 bytes in 378 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:03.535698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d61264-f870-430c-a546-d3551acea7ba None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:03.536176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d61264-f870-430c-a546-d3551acea7ba None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:03.536265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d61264-f870-430c-a546-d3551acea7ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:03.557453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-81d61264-f870-430c-a546-d3551acea7ba None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:03.558676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-81d61264-f870-430c-a546-d3551acea7ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:03.620435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 810/1623] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:03.628385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96efa9a9-aa9b-46f5-83c1-43abc51632ee None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:03.629089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96efa9a9-aa9b-46f5-83c1-43abc51632ee None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:03.633295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96efa9a9-aa9b-46f5-83c1-43abc51632ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:03.650225 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-96efa9a9-aa9b-46f5-83c1-43abc51632ee None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:03.651481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-96efa9a9-aa9b-46f5-83c1-43abc51632ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:03.659372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 814/1624] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:03.664591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d61264-f870-430c-a546-d3551acea7ba None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:03.674594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d61264-f870-430c-a546-d3551acea7ba None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:03.680841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:03.681475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:03.681740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:03.682014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:03.682688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:03.682992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:03.684593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:03.696744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 811/1625] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:03:03.701087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96efa9a9-aa9b-46f5-83c1-43abc51632ee None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:03.718000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96efa9a9-aa9b-46f5-83c1-43abc51632ee None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:03.726529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6905eac7-ed1b-4ce0-bc37-859333faea07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:03.727441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6905eac7-ed1b-4ce0-bc37-859333faea07 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:03.731034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6905eac7-ed1b-4ce0-bc37-859333faea07 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:03.731034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6905eac7-ed1b-4ce0-bc37-859333faea07 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:03.731034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6905eac7-ed1b-4ce0-bc37-859333faea07 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:03.731034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6905eac7-ed1b-4ce0-bc37-859333faea07 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:03.850997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 815/1626] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:03 2022] POST /identity/v3/projects => generated 369 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:03.855703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:03.866106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c65354f-cd3b-4130-b79f-c59dd7e105ad admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:03.929398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf60690-d5e1-4227-931a-acf024e5c750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:03.930040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf60690-d5e1-4227-931a-acf024e5c750 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:03.930325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf60690-d5e1-4227-931a-acf024e5c750 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:03.935850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf60690-d5e1-4227-931a-acf024e5c750 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:03.935850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf60690-d5e1-4227-931a-acf024e5c750 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:03.935850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf60690-d5e1-4227-931a-acf024e5c750 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:04.050420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 812/1627] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:03 2022] POST /identity/v3/users => generated 285 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:04.051788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6905eac7-ed1b-4ce0-bc37-859333faea07 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:04.062744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6905eac7-ed1b-4ce0-bc37-859333faea07 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:04.067475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e025f6d-6a48-42c5-8bff-e436f2437f5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:04.068614 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e025f6d-6a48-42c5-8bff-e436f2437f5c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:04.068904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e025f6d-6a48-42c5-8bff-e436f2437f5c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:04.070002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e025f6d-6a48-42c5-8bff-e436f2437f5c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:04.070201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e025f6d-6a48-42c5-8bff-e436f2437f5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:04.071835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e025f6d-6a48-42c5-8bff-e436f2437f5c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:04.085115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 816/1628] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:03:04.100414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf60690-d5e1-4227-931a-acf024e5c750 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:04.118120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf60690-d5e1-4227-931a-acf024e5c750 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:04.125737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7175c43-5e60-42f5-b1fd-b93d4b443fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:04.126466 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7175c43-5e60-42f5-b1fd-b93d4b443fc5 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:04.126798 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7175c43-5e60-42f5-b1fd-b93d4b443fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:04.127119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7175c43-5e60-42f5-b1fd-b93d4b443fc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:04.127793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7175c43-5e60-42f5-b1fd-b93d4b443fc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:04.129400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7175c43-5e60-42f5-b1fd-b93d4b443fc5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:04.142012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 813/1629] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:04 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:04.146848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e025f6d-6a48-42c5-8bff-e436f2437f5c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:04.157881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e025f6d-6a48-42c5-8bff-e436f2437f5c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:04.162753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:04.163513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:04.163795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:04.164066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] PATH_INFO: `/v3/projects/31156c69a6ea49799ae059f8e611da44/users/e4c00ad1a2544535adc54e60f7749e99/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:04.181099 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] RBAC: Authorizing `identity:create_grant(project_id=31156c69a6ea49799ae059f8e611da44, user_id=e4c00ad1a2544535adc54e60f7749e99, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:04.183850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:04.263999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 817/1630] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:04 2022] PUT /identity/v3/projects/31156c69a6ea49799ae059f8e611da44/users/e4c00ad1a2544535adc54e60f7749e99/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:04.276172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7969f070-1abf-4fb2-b609-7a951c873b79 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:04.276452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7969f070-1abf-4fb2-b609-7a951c873b79 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:04.276917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7969f070-1abf-4fb2-b609-7a951c873b79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:04.305115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-7969f070-1abf-4fb2-b609-7a951c873b79 None None] MFA Rules not processed for user `e4c00ad1a2544535adc54e60f7749e99`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:04.306442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7969f070-1abf-4fb2-b609-7a951c873b79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:04.320297 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:04.378918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 814/1631] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:04 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:04.387432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:04.389146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:04.445300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:04.474481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7969f070-1abf-4fb2-b609-7a951c873b79 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:04.475933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7969f070-1abf-4fb2-b609-7a951c873b79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:04.498497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8934adbc-1a03-408f-a944-7f123ee2259c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:04.503844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5996d17e-669b-40d1-a9e9-4e69fddda142 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:04.504658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5996d17e-669b-40d1-a9e9-4e69fddda142 service glance] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:04.512660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5996d17e-669b-40d1-a9e9-4e69fddda142 service glance] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:04.512660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5996d17e-669b-40d1-a9e9-4e69fddda142 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:04.516140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5996d17e-669b-40d1-a9e9-4e69fddda142 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:04.517695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5996d17e-669b-40d1-a9e9-4e69fddda142 service glance] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:04.528310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 818/1632] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:04 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:04.566999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7969f070-1abf-4fb2-b609-7a951c873b79 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:04.572062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:04.572776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:04.573237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:04.573528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] PATH_INFO: `/v3/users/0116b22b9dd143d3b74b9b56989cd7fa` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:04.574738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0116b22b9dd143d3b74b9b56989cd7fa)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:04.576749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:04.790797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0116b22b9dd143d3b74b9b56989cd7fa'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:04.858214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0116b22b9dd143d3b74b9b56989cd7fa'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:04.871698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0116b22b9dd143d3b74b9b56989cd7fa'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:04.879116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0116b22b9dd143d3b74b9b56989cd7fa'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:04.880091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0116b22b9dd143d3b74b9b56989cd7fa'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:04.890037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 815/1633] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:04 2022] DELETE /identity/v3/users/0116b22b9dd143d3b74b9b56989cd7fa => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:04.891279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d3e4fb-f3bf-45db-bf63-767efa5c82ba None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:04.891562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d3e4fb-f3bf-45db-bf63-767efa5c82ba None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:04.892026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d3e4fb-f3bf-45db-bf63-767efa5c82ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:04.912850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-99d3e4fb-f3bf-45db-bf63-767efa5c82ba None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:04.914093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-99d3e4fb-f3bf-45db-bf63-767efa5c82ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:04.986286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 819/1634] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:04.998468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:05.002051 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:05.047319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:05.065065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a85f5e-ab2f-43ed-ab0e-57f680d57255 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:05.068850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613245d4-0183-4b32-817f-61094701538a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:05.069464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613245d4-0183-4b32-817f-61094701538a service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:05.069745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613245d4-0183-4b32-817f-61094701538a service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:05.070020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613245d4-0183-4b32-817f-61094701538a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:05.080401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613245d4-0183-4b32-817f-61094701538a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:05.081845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613245d4-0183-4b32-817f-61094701538a service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:05.091704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 816/1635] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:05.678785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d3e4fb-f3bf-45db-bf63-767efa5c82ba None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:05.680399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-99d3e4fb-f3bf-45db-bf63-767efa5c82ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:05.696529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d3e4fb-f3bf-45db-bf63-767efa5c82ba None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:05.704922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:05.705569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:05.705947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:05.706354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] PATH_INFO: `/v3/projects/90c65ab417f14b209fe154f3e9e74516` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:05.707426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90c65ab417f14b209fe154f3e9e74516)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:05.710338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:05.827404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90c65ab417f14b209fe154f3e9e74516', 'request_id': 'req-c30b716b-620d-4f25-978d-68678d02ace4'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:05.833714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90c65ab417f14b209fe154f3e9e74516', 'request_id': 'req-c30b716b-620d-4f25-978d-68678d02ace4'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:05.839979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90c65ab417f14b209fe154f3e9e74516', 'request_id': 'req-c30b716b-620d-4f25-978d-68678d02ace4'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:05.928208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] The token cache is being invalidate because project 90c65ab417f14b209fe154f3e9e74516 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:05.928557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:05.929210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-c30b716b-620d-4f25-978d-68678d02ace4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:05.930707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 820/1636] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:05 2022] DELETE /identity/v3/projects/90c65ab417f14b209fe154f3e9e74516 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:05.935647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efbb769-742e-415f-b93e-3837a24a9d2d None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:05.936130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efbb769-742e-415f-b93e-3837a24a9d2d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:05.936130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efbb769-742e-415f-b93e-3837a24a9d2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:05.956568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5efbb769-742e-415f-b93e-3837a24a9d2d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:05.957774 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5efbb769-742e-415f-b93e-3837a24a9d2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:06.019377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 817/1637] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:06.026960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123b05-7fc2-4220-ac63-dc8c3c5565b6 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.027251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123b05-7fc2-4220-ac63-dc8c3c5565b6 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.027525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc123b05-7fc2-4220-ac63-dc8c3c5565b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.047251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-cc123b05-7fc2-4220-ac63-dc8c3c5565b6 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:06.048531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cc123b05-7fc2-4220-ac63-dc8c3c5565b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:06.057343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 821/1638] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:06.061600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efbb769-742e-415f-b93e-3837a24a9d2d None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:06.072330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efbb769-742e-415f-b93e-3837a24a9d2d None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:06.077053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:06.077655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.077942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.078218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.078874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:06.079212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:06.080835 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:06.094103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 818/1639] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:03:06.098676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc123b05-7fc2-4220-ac63-dc8c3c5565b6 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:06.108558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc123b05-7fc2-4220-ac63-dc8c3c5565b6 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:06.114410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d19f864-7acf-462b-8ade-c4096aa8eec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:06.115556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19f864-7acf-462b-8ade-c4096aa8eec1 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.115834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19f864-7acf-462b-8ade-c4096aa8eec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.116126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d19f864-7acf-462b-8ade-c4096aa8eec1 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.116768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d19f864-7acf-462b-8ade-c4096aa8eec1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:06.118696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d19f864-7acf-462b-8ade-c4096aa8eec1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:06.198100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 822/1640] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:06 2022] POST /identity/v3/projects => generated 383 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:06.202070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:06.226725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238d0a50-110d-4806-bf86-8b99d5b13492 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:06.299245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf89a080-677f-4c73-9ad5-e54aa7813a1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:06.299857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89a080-677f-4c73-9ad5-e54aa7813a1d admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.300138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89a080-677f-4c73-9ad5-e54aa7813a1d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.300440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89a080-677f-4c73-9ad5-e54aa7813a1d admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.301214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf89a080-677f-4c73-9ad5-e54aa7813a1d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:06.303762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf89a080-677f-4c73-9ad5-e54aa7813a1d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:06.422380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 819/1641] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:06 2022] POST /identity/v3/users => generated 292 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:06.426362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d19f864-7acf-462b-8ade-c4096aa8eec1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:06.452713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d19f864-7acf-462b-8ade-c4096aa8eec1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:06.457677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ae6c2d-6ebc-46b8-9689-d098f235b32c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:06.458295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ae6c2d-6ebc-46b8-9689-d098f235b32c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.458633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ae6c2d-6ebc-46b8-9689-d098f235b32c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.458947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ae6c2d-6ebc-46b8-9689-d098f235b32c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.459727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ae6c2d-6ebc-46b8-9689-d098f235b32c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:06.461421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ae6c2d-6ebc-46b8-9689-d098f235b32c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:06.476842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 823/1642] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:06 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:06.478280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf89a080-677f-4c73-9ad5-e54aa7813a1d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:06.488903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf89a080-677f-4c73-9ad5-e54aa7813a1d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:06.494270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00293d13-49be-4fba-81dc-85705fa01379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:06.494942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00293d13-49be-4fba-81dc-85705fa01379 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.495320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00293d13-49be-4fba-81dc-85705fa01379 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.495599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00293d13-49be-4fba-81dc-85705fa01379 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.496244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00293d13-49be-4fba-81dc-85705fa01379 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:06.497854 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00293d13-49be-4fba-81dc-85705fa01379 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:06.507896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 820/1643] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:06.511999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ae6c2d-6ebc-46b8-9689-d098f235b32c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:06.522139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ae6c2d-6ebc-46b8-9689-d098f235b32c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:06.526661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:06.527339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.527616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.527891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] PATH_INFO: `/v3/projects/6ec4ff60137544109a46d3a5810251f5/users/7ea5afc217f843389fe051b42616e09d/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.536251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:06.543438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ec4ff60137544109a46d3a5810251f5, user_id=7ea5afc217f843389fe051b42616e09d, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:06.546084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:06.631944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 824/1644] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:06 2022] PUT /identity/v3/projects/6ec4ff60137544109a46d3a5810251f5/users/7ea5afc217f843389fe051b42616e09d/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:06.637350 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861f074-cbbb-4901-93d0-8430373ceef4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.637858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861f074-cbbb-4901-93d0-8430373ceef4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.638138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a861f074-cbbb-4901-93d0-8430373ceef4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.661658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a861f074-cbbb-4901-93d0-8430373ceef4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:06.663613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a861f074-cbbb-4901-93d0-8430373ceef4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:06.706169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:06.726287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 821/1645] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:06.736278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:06.746225 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:06.820466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05863e92-834f-45d9-bbbf-03d585c468a6 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:06.824718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:06.825324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:06.825601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:06.825871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:06.837048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:06.838522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:06.849221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 825/1646] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:08.604008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf94da5-a218-4a9c-8da3-f6c3b2e9e4ad None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:08.604008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf94da5-a218-4a9c-8da3-f6c3b2e9e4ad None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:08.604008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf94da5-a218-4a9c-8da3-f6c3b2e9e4ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:08.634844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-faf94da5-a218-4a9c-8da3-f6c3b2e9e4ad None None] MFA Rules not processed for user `7ea5afc217f843389fe051b42616e09d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:08.636487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-faf94da5-a218-4a9c-8da3-f6c3b2e9e4ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:08.723068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 822/1647] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:08.735825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:08.741069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:08.812678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87016c3e-ac40-4315-bc0d-7f4383802058 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:08.820739 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:08.820739 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:08.820739 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:08.820739 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:08.827984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:08.829474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:08.839082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 826/1648] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:08 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:11.649216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa4794-b999-487a-9d16-b0dcc5c85860 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:11.649768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa4794-b999-487a-9d16-b0dcc5c85860 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:11.649768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fa4794-b999-487a-9d16-b0dcc5c85860 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:11.687996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-30fa4794-b999-487a-9d16-b0dcc5c85860 None None] MFA Rules not processed for user `7ea5afc217f843389fe051b42616e09d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:11.689377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-30fa4794-b999-487a-9d16-b0dcc5c85860 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:11.708132 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 823/1649] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:11 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:14.421951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:14.423534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:14.437721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0d0a56-d522-4b17-8a99-1715f8475e14 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:14.442305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:14.444449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:14.444449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:14.444449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] PATH_INFO: `/v3/users/3bb0c59ee869439798d6bbcfac57706d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:14.445456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bb0c59ee869439798d6bbcfac57706d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:14.447722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:14.648366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bb0c59ee869439798d6bbcfac57706d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:14.649374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bb0c59ee869439798d6bbcfac57706d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:14.669336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bb0c59ee869439798d6bbcfac57706d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:14.681569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bb0c59ee869439798d6bbcfac57706d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:14.683421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bb0c59ee869439798d6bbcfac57706d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:14.807049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 827/1650] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:14 2022] DELETE /identity/v3/users/3bb0c59ee869439798d6bbcfac57706d => generated 0 bytes in 386 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:14.811855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce839f07-ad85-4583-a119-911cc166db67 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:14.812365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce839f07-ad85-4583-a119-911cc166db67 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:14.812452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce839f07-ad85-4583-a119-911cc166db67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:14.832230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ce839f07-ad85-4583-a119-911cc166db67 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:14.833461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ce839f07-ad85-4583-a119-911cc166db67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:14.910658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 824/1651] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:14.920444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:14.921937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:14.929520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:14.981305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a842e44-146f-4559-a653-0bdac91d2c04 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:14.985023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d493b8b5-69a4-4034-a3b4-acf2742f9642 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:14.985632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d493b8b5-69a4-4034-a3b4-acf2742f9642 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:14.985911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d493b8b5-69a4-4034-a3b4-acf2742f9642 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:14.986184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d493b8b5-69a4-4034-a3b4-acf2742f9642 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:14.996640 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d493b8b5-69a4-4034-a3b4-acf2742f9642 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:14.998142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d493b8b5-69a4-4034-a3b4-acf2742f9642 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:15.008486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 828/1652] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:15.544079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce839f07-ad85-4583-a119-911cc166db67 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:15.548736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ce839f07-ad85-4583-a119-911cc166db67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:15.567458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce839f07-ad85-4583-a119-911cc166db67 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:15.575205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:15.575864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:15.576144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:15.576416 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] PATH_INFO: `/v3/projects/3aa3c9449d0346cc8a0917a487274123` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:15.585611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3aa3c9449d0346cc8a0917a487274123)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:15.585611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:15.868186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3aa3c9449d0346cc8a0917a487274123', 'request_id': 'req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:15.956226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3aa3c9449d0346cc8a0917a487274123', 'request_id': 'req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:15.967214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:15.968672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3aa3c9449d0346cc8a0917a487274123', 'request_id': 'req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:15.981003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] The token cache is being invalidate because project 3aa3c9449d0346cc8a0917a487274123 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:15.981301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:15.981989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:15.983559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 825/1653] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:15 2022] DELETE /identity/v3/projects/3aa3c9449d0346cc8a0917a487274123 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:15.987835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d493b8b5-69a4-4034-a3b4-acf2742f9642 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:15.998347 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d493b8b5-69a4-4034-a3b4-acf2742f9642 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:16.058601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:16.059159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:16.059562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:16.059837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] PATH_INFO: `/v3/users/6709098dab7d4dbf9da2a8f0bde7ff06` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:16.060958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=6709098dab7d4dbf9da2a8f0bde7ff06)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:16.062890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:16.227092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6709098dab7d4dbf9da2a8f0bde7ff06'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:16.227746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6709098dab7d4dbf9da2a8f0bde7ff06'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:16.240397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6709098dab7d4dbf9da2a8f0bde7ff06'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:16.247356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6709098dab7d4dbf9da2a8f0bde7ff06'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:16.247997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-14601815-3d0f-446b-92ff-e907161b34eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6709098dab7d4dbf9da2a8f0bde7ff06'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:16.385215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 829/1654] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:15 2022] DELETE /identity/v3/users/6709098dab7d4dbf9da2a8f0bde7ff06 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:16.832260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:16.834235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:16.846822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f31b071-bcca-48f3-a6a2-9ecb9eec9716 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:16.903102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:16.903812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:16.904046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:16.904318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] PATH_INFO: `/v3/projects/c63a114f1bca481984136b9ba64c58a0` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:16.905437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c63a114f1bca481984136b9ba64c58a0)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:16.907398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:17.041516 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c63a114f1bca481984136b9ba64c58a0', 'request_id': 'req-b77dd40d-74e1-40d2-b221-7e84a372cdf7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:17.116370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c63a114f1bca481984136b9ba64c58a0', 'request_id': 'req-b77dd40d-74e1-40d2-b221-7e84a372cdf7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:17.122108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c63a114f1bca481984136b9ba64c58a0', 'request_id': 'req-b77dd40d-74e1-40d2-b221-7e84a372cdf7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:17.134025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] The token cache is being invalidate because project c63a114f1bca481984136b9ba64c58a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:17.134308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:17.134932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b77dd40d-74e1-40d2-b221-7e84a372cdf7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:17.136440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 826/1655] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:16 2022] DELETE /identity/v3/projects/c63a114f1bca481984136b9ba64c58a0 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:17.141357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032fade-6800-4277-9640-025345fe081c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.141830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032fade-6800-4277-9640-025345fe081c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.141926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2032fade-6800-4277-9640-025345fe081c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.161336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2032fade-6800-4277-9640-025345fe081c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:17.162605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2032fade-6800-4277-9640-025345fe081c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:17.184261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:17.229992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 830/1656] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:17.231941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823c56b0-b317-4f26-ac81-5704eddd1e3e None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.232237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823c56b0-b317-4f26-ac81-5704eddd1e3e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.235926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823c56b0-b317-4f26-ac81-5704eddd1e3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.265118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-823c56b0-b317-4f26-ac81-5704eddd1e3e None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:17.266514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-823c56b0-b317-4f26-ac81-5704eddd1e3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:17.278585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 827/1657] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:17.280051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2032fade-6800-4277-9640-025345fe081c None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:17.289638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2032fade-6800-4277-9640-025345fe081c None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:17.294207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:17.297087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.297087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.297087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.297087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:17.297087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:17.298203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:17.310333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 831/1658] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:03:17.314703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823c56b0-b317-4f26-ac81-5704eddd1e3e None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:17.325776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823c56b0-b317-4f26-ac81-5704eddd1e3e None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:17.329497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ae42a-aa2f-490b-b2f3-d6d08a26bf67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:17.330095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ae42a-aa2f-490b-b2f3-d6d08a26bf67 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.330370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ae42a-aa2f-490b-b2f3-d6d08a26bf67 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.330641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03ae42a-aa2f-490b-b2f3-d6d08a26bf67 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.331316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03ae42a-aa2f-490b-b2f3-d6d08a26bf67 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:17.333443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03ae42a-aa2f-490b-b2f3-d6d08a26bf67 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:17.437899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 828/1659] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:17 2022] POST /identity/v3/projects => generated 379 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:17.441839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:17.452558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adaa4ea4-7560-4d66-8690-74d29ddae6b2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:17.507381 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea821f3-c6e3-4ce7-8aa0-f9f1ff02bd0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:17.507983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea821f3-c6e3-4ce7-8aa0-f9f1ff02bd0a admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.508263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea821f3-c6e3-4ce7-8aa0-f9f1ff02bd0a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.508538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea821f3-c6e3-4ce7-8aa0-f9f1ff02bd0a admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.509169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea821f3-c6e3-4ce7-8aa0-f9f1ff02bd0a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:17.511094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea821f3-c6e3-4ce7-8aa0-f9f1ff02bd0a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:17.593315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 832/1660] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:17 2022] POST /identity/v3/users => generated 290 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:17.597398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ae42a-aa2f-490b-b2f3-d6d08a26bf67 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:17.607757 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03ae42a-aa2f-490b-b2f3-d6d08a26bf67 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:17.612369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4c3332-d6b2-49b8-8469-465151244c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:17.612957 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c3332-d6b2-49b8-8469-465151244c81 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.613232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c3332-d6b2-49b8-8469-465151244c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.613501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c3332-d6b2-49b8-8469-465151244c81 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.614145 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4c3332-d6b2-49b8-8469-465151244c81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:17.616076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4c3332-d6b2-49b8-8469-465151244c81 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:17.629161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 829/1661] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:17.633971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea821f3-c6e3-4ce7-8aa0-f9f1ff02bd0a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:17.644446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea821f3-c6e3-4ce7-8aa0-f9f1ff02bd0a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:17.649709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74a61a1-c7ad-4a61-9434-85683fac600c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:17.650303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74a61a1-c7ad-4a61-9434-85683fac600c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.650581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74a61a1-c7ad-4a61-9434-85683fac600c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.650852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74a61a1-c7ad-4a61-9434-85683fac600c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.651528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74a61a1-c7ad-4a61-9434-85683fac600c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:17.653140 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74a61a1-c7ad-4a61-9434-85683fac600c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:17.663936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 833/1662] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:17.668126 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4c3332-d6b2-49b8-8469-465151244c81 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:17.677832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4c3332-d6b2-49b8-8469-465151244c81 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:17.682760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7ce848-78ca-494c-b94f-3acf5b38df93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:17.683426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ce848-78ca-494c-b94f-3acf5b38df93 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.683703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ce848-78ca-494c-b94f-3acf5b38df93 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.683974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7ce848-78ca-494c-b94f-3acf5b38df93 admin admin] PATH_INFO: `/v3/projects/1ce74ef02cf94741bd52d88470722de0/users/89369d1a91044136afa32521d6321433/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.698797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7ce848-78ca-494c-b94f-3acf5b38df93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ce74ef02cf94741bd52d88470722de0, user_id=89369d1a91044136afa32521d6321433, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:17.701472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7ce848-78ca-494c-b94f-3acf5b38df93 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:17.808475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 830/1663] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:17 2022] PUT /identity/v3/projects/1ce74ef02cf94741bd52d88470722de0/users/89369d1a91044136afa32521d6321433/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:17.819448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548a3e1-f9a9-405a-9397-454bc03a87f8 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:17.819825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548a3e1-f9a9-405a-9397-454bc03a87f8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:17.820101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f548a3e1-f9a9-405a-9397-454bc03a87f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:17.848172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f548a3e1-f9a9-405a-9397-454bc03a87f8 None None] MFA Rules not processed for user `89369d1a91044136afa32521d6321433`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:17.849374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f548a3e1-f9a9-405a-9397-454bc03a87f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:17.911448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 834/1664] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:17 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:17.916015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7ce848-78ca-494c-b94f-3acf5b38df93 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:17.940789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:17.993563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7ce848-78ca-494c-b94f-3acf5b38df93 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:17.998792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fb8c84-9c48-4667-bb22-bedd4ece0ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.001210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fb8c84-9c48-4667-bb22-bedd4ece0ea3 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.001210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fb8c84-9c48-4667-bb22-bedd4ece0ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.001210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fb8c84-9c48-4667-bb22-bedd4ece0ea3 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.001210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fb8c84-9c48-4667-bb22-bedd4ece0ea3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:18.002587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fb8c84-9c48-4667-bb22-bedd4ece0ea3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:18.095215 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 831/1665] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:17 2022] POST /identity/v3/projects => generated 377 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:18.099268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f548a3e1-f9a9-405a-9397-454bc03a87f8 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:18.109882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f548a3e1-f9a9-405a-9397-454bc03a87f8 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:18.127793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:18.171603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94001eee-282b-450d-b244-b79e9d9b3204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.172306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94001eee-282b-450d-b244-b79e9d9b3204 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.172495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94001eee-282b-450d-b244-b79e9d9b3204 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.172920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94001eee-282b-450d-b244-b79e9d9b3204 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.173574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94001eee-282b-450d-b244-b79e9d9b3204 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:18.175527 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94001eee-282b-450d-b244-b79e9d9b3204 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:18.271337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 835/1666] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:18 2022] POST /identity/v3/users => generated 289 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:18.276027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fb8c84-9c48-4667-bb22-bedd4ece0ea3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:18.286518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fb8c84-9c48-4667-bb22-bedd4ece0ea3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:18.291392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddee8264-26f6-4dd0-9e67-963db02bef3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.295054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddee8264-26f6-4dd0-9e67-963db02bef3a admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.295054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddee8264-26f6-4dd0-9e67-963db02bef3a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.295054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddee8264-26f6-4dd0-9e67-963db02bef3a admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.295054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddee8264-26f6-4dd0-9e67-963db02bef3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:18.295361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddee8264-26f6-4dd0-9e67-963db02bef3a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:18.306013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 832/1667] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:18.309789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94001eee-282b-450d-b244-b79e9d9b3204 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:18.320264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94001eee-282b-450d-b244-b79e9d9b3204 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:18.325278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f99d26-81f4-4695-94c7-17d373e17dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.325875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f99d26-81f4-4695-94c7-17d373e17dac admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.326202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f99d26-81f4-4695-94c7-17d373e17dac admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.326481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f99d26-81f4-4695-94c7-17d373e17dac admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.327170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f99d26-81f4-4695-94c7-17d373e17dac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:18.328804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f99d26-81f4-4695-94c7-17d373e17dac admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:18.339674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 836/1668] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:18.343824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddee8264-26f6-4dd0-9e67-963db02bef3a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:18.354088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddee8264-26f6-4dd0-9e67-963db02bef3a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:18.358784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8881f685-046d-4b40-8d92-c6154ee5e63f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.359592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8881f685-046d-4b40-8d92-c6154ee5e63f admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.359937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8881f685-046d-4b40-8d92-c6154ee5e63f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.360314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8881f685-046d-4b40-8d92-c6154ee5e63f admin admin] PATH_INFO: `/v3/projects/e2411fefbf584b849a3096e20d1351de/users/c57b5a9e2c324023ab46127d5827bf6b/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.375961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8881f685-046d-4b40-8d92-c6154ee5e63f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2411fefbf584b849a3096e20d1351de, user_id=c57b5a9e2c324023ab46127d5827bf6b, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:18.378667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8881f685-046d-4b40-8d92-c6154ee5e63f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:18.479373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 833/1669] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:18 2022] PUT /identity/v3/projects/e2411fefbf584b849a3096e20d1351de/users/c57b5a9e2c324023ab46127d5827bf6b/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:18.487821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace84a55-efc2-4c0b-b6af-db27f5615631 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.488109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace84a55-efc2-4c0b-b6af-db27f5615631 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.488389 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace84a55-efc2-4c0b-b6af-db27f5615631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.523310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ace84a55-efc2-4c0b-b6af-db27f5615631 None None] MFA Rules not processed for user `c57b5a9e2c324023ab46127d5827bf6b`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:18.524718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ace84a55-efc2-4c0b-b6af-db27f5615631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:18.590775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 837/1670] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:18.595270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8881f685-046d-4b40-8d92-c6154ee5e63f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:18.657114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8881f685-046d-4b40-8d92-c6154ee5e63f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:18.661795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac139ad1-f10e-4da5-a6ef-9c8d57c277b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.662395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac139ad1-f10e-4da5-a6ef-9c8d57c277b6 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.662672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac139ad1-f10e-4da5-a6ef-9c8d57c277b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.662943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac139ad1-f10e-4da5-a6ef-9c8d57c277b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.663598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac139ad1-f10e-4da5-a6ef-9c8d57c277b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:18.665527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac139ad1-f10e-4da5-a6ef-9c8d57c277b6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:18.745737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 834/1671] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:18 2022] POST /identity/v3/projects => generated 377 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:18.750002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace84a55-efc2-4c0b-b6af-db27f5615631 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:18.760658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace84a55-efc2-4c0b-b6af-db27f5615631 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:18.820463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0787eb3d-e277-4357-a554-e69ae9952b98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.821074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0787eb3d-e277-4357-a554-e69ae9952b98 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.821351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0787eb3d-e277-4357-a554-e69ae9952b98 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.821618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0787eb3d-e277-4357-a554-e69ae9952b98 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.822262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0787eb3d-e277-4357-a554-e69ae9952b98 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:18.824308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0787eb3d-e277-4357-a554-e69ae9952b98 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:18.927654 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 838/1672] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:18 2022] POST /identity/v3/users => generated 295 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:18.932090 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac139ad1-f10e-4da5-a6ef-9c8d57c277b6 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:18.943072 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac139ad1-f10e-4da5-a6ef-9c8d57c277b6 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:18.948049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3955a91-7260-4750-bf6a-21d9ddb32d0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.948648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3955a91-7260-4750-bf6a-21d9ddb32d0b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.948924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3955a91-7260-4750-bf6a-21d9ddb32d0b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.949194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3955a91-7260-4750-bf6a-21d9ddb32d0b admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.949836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3955a91-7260-4750-bf6a-21d9ddb32d0b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:18.951460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3955a91-7260-4750-bf6a-21d9ddb32d0b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:18.962301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 835/1673] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:18.966558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0787eb3d-e277-4357-a554-e69ae9952b98 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:18.977258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0787eb3d-e277-4357-a554-e69ae9952b98 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:18.981763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f167a10-b5c3-464b-adfb-c93423e608be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:18.982414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f167a10-b5c3-464b-adfb-c93423e608be admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:18.982697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f167a10-b5c3-464b-adfb-c93423e608be admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:18.982999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f167a10-b5c3-464b-adfb-c93423e608be admin admin] PATH_INFO: `/v3/projects/405ca0c3f13b4bbfaace1e39c454d101/users/b08dd7da0eba4ddba43301c23cbc9667/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:18.997446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f167a10-b5c3-464b-adfb-c93423e608be admin admin] RBAC: Authorizing `identity:create_grant(project_id=405ca0c3f13b4bbfaace1e39c454d101, user_id=b08dd7da0eba4ddba43301c23cbc9667, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:19.000090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f167a10-b5c3-464b-adfb-c93423e608be admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:19.003192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:19.063662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 839/1674] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:18 2022] PUT /identity/v3/projects/405ca0c3f13b4bbfaace1e39c454d101/users/b08dd7da0eba4ddba43301c23cbc9667/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:19.067776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3955a91-7260-4750-bf6a-21d9ddb32d0b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:19.078400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3955a91-7260-4750-bf6a-21d9ddb32d0b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:19.121628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:19.147378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85685c1-7b39-4dd7-b41d-b44a8d5d6232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:19.148087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85685c1-7b39-4dd7-b41d-b44a8d5d6232 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:19.148468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85685c1-7b39-4dd7-b41d-b44a8d5d6232 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:19.148805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85685c1-7b39-4dd7-b41d-b44a8d5d6232 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:19.149470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85685c1-7b39-4dd7-b41d-b44a8d5d6232 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:19.151141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85685c1-7b39-4dd7-b41d-b44a8d5d6232 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:19.162226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 836/1675] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:19 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:19.174921 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f167a10-b5c3-464b-adfb-c93423e608be admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:19.186765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f167a10-b5c3-464b-adfb-c93423e608be admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:19.191554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:19.192205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:19.194832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:19.194832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] PATH_INFO: `/v3/projects/405ca0c3f13b4bbfaace1e39c454d101/users/b08dd7da0eba4ddba43301c23cbc9667/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:19.194832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=405ca0c3f13b4bbfaace1e39c454d101, user_id=b08dd7da0eba4ddba43301c23cbc9667, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:19.197350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:19.208045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 840/1676] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:19 2022] PUT /identity/v3/projects/405ca0c3f13b4bbfaace1e39c454d101/users/b08dd7da0eba4ddba43301c23cbc9667/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:19.218414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c282bc2-0601-40a6-8e95-19630279a8fc None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:19.218707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c282bc2-0601-40a6-8e95-19630279a8fc None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:19.219013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c282bc2-0601-40a6-8e95-19630279a8fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:19.257200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3c282bc2-0601-40a6-8e95-19630279a8fc None None] MFA Rules not processed for user `b08dd7da0eba4ddba43301c23cbc9667`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:19.258705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3c282bc2-0601-40a6-8e95-19630279a8fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:19.328258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 837/1677] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:19 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:19.340018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:19.341699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:19.399862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2bf5b1-7a9e-4949-8785-cbd9c8bae0c3 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:19.403471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119c65dd-d80e-40fd-973b-53a9ab3c5149 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:19.404072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c65dd-d80e-40fd-973b-53a9ab3c5149 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:19.404355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c65dd-d80e-40fd-973b-53a9ab3c5149 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:19.404630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119c65dd-d80e-40fd-973b-53a9ab3c5149 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:19.413518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119c65dd-d80e-40fd-973b-53a9ab3c5149 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:19.414975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119c65dd-d80e-40fd-973b-53a9ab3c5149 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:19.424290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 841/1678] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:19 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:19.560381 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c282bc2-0601-40a6-8e95-19630279a8fc None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:19.570989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c282bc2-0601-40a6-8e95-19630279a8fc None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:19.574908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:19.575628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:19.575628 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:19.575801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:19.665338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:19.666931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:19.680133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 838/1679] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:19 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:23.913084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119c65dd-d80e-40fd-973b-53a9ab3c5149 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:23.975708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119c65dd-d80e-40fd-973b-53a9ab3c5149 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:23.980823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:23.981453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:23.981730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:23.982118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] PATH_INFO: `/v3/users/89369d1a91044136afa32521d6321433` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:23.983235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] RBAC: Authorizing `identity:delete_user(user_id=89369d1a91044136afa32521d6321433)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:23.985467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:24.086600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:24.088052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:24.102763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db114b8-9ac2-4517-862e-60aa33061d2e service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:24.107831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:24.108444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:24.108720 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:24.108987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] PATH_INFO: `/v3/users/5deec494f31646fb93b3e66e32def88a` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:24.110092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5deec494f31646fb93b3e66e32def88a)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:24.112108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:24.165447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89369d1a91044136afa32521d6321433'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.166108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89369d1a91044136afa32521d6321433'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.179648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89369d1a91044136afa32521d6321433'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.186632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89369d1a91044136afa32521d6321433'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.187590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3a409d0f-d039-4b64-8648-cd43339c490b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89369d1a91044136afa32521d6321433'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.271273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 842/1680] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:23 2022] DELETE /identity/v3/users/89369d1a91044136afa32521d6321433 => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:24.280431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bb4f5f-4dc9-4f99-a377-683d14a23dc0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:24.280718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bb4f5f-4dc9-4f99-a377-683d14a23dc0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:24.281014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bb4f5f-4dc9-4f99-a377-683d14a23dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:24.290139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:24.306335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-95bb4f5f-4dc9-4f99-a377-683d14a23dc0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:24.306335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-95bb4f5f-4dc9-4f99-a377-683d14a23dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:24.337176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5deec494f31646fb93b3e66e32def88a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.367025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 843/1681] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:24.378461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bb4f5f-4dc9-4f99-a377-683d14a23dc0 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:24.379893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-95bb4f5f-4dc9-4f99-a377-683d14a23dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:24.413366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5deec494f31646fb93b3e66e32def88a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.416071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:24.430089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5deec494f31646fb93b3e66e32def88a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.438109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5deec494f31646fb93b3e66e32def88a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.439070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3c0c8cca-9e5c-4a5c-a65c-7487697bfb52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5deec494f31646fb93b3e66e32def88a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:24.445304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 839/1682] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:24 2022] DELETE /identity/v3/users/5deec494f31646fb93b3e66e32def88a => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:24.449641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0351f7-8fe7-496c-8cab-2fecef151c3f None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:24.450103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0351f7-8fe7-496c-8cab-2fecef151c3f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:24.450385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0351f7-8fe7-496c-8cab-2fecef151c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:24.452507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bb4f5f-4dc9-4f99-a377-683d14a23dc0 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:24.471096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-fe0351f7-8fe7-496c-8cab-2fecef151c3f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:24.472389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fe0351f7-8fe7-496c-8cab-2fecef151c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:24.502502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:24.503279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:24.503607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:24.503887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:24.505622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:24.533147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 840/1683] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:24.543401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0351f7-8fe7-496c-8cab-2fecef151c3f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:24.545547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fe0351f7-8fe7-496c-8cab-2fecef151c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:24.557265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0351f7-8fe7-496c-8cab-2fecef151c3f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:24.561016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:24.561620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:24.561897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:24.562169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:24.569022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:24.570546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:24.571332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:24.573437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:24.580837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 844/1684] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:24.583434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 841/1685] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:25.323598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:25.325113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:25.336990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be56d8fd-3fa3-4c2b-872f-b6d4994ae56f service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:25.348437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:25.350132 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:25.358561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:25.359716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:25.360001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:25.360354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] PATH_INFO: `/v3/projects/1ce74ef02cf94741bd52d88470722de0` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:25.368476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ce74ef02cf94741bd52d88470722de0)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:25.368476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:25.373755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bd1391-0f71-4052-bdfb-e85fe6d73954 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:25.376049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:25.389952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:25.390741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:25.391113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:25.391333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] PATH_INFO: `/v3/projects/9d1926d34e824016927d5d2eeefca945` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:25.397676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d1926d34e824016927d5d2eeefca945)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:25.402506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:25.491245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ce74ef02cf94741bd52d88470722de0', 'request_id': 'req-49336c02-9c08-4a7c-8d84-6243a489ae0d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.502274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ce74ef02cf94741bd52d88470722de0', 'request_id': 'req-49336c02-9c08-4a7c-8d84-6243a489ae0d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.511045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ce74ef02cf94741bd52d88470722de0', 'request_id': 'req-49336c02-9c08-4a7c-8d84-6243a489ae0d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.530167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d1926d34e824016927d5d2eeefca945', 'request_id': 'req-ecf1315b-d509-43e7-89fe-70bbb26dce7f'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.620676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] The token cache is being invalidate because project 1ce74ef02cf94741bd52d88470722de0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:25.621189 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.621996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.623957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 845/1686] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:25 2022] DELETE /identity/v3/projects/1ce74ef02cf94741bd52d88470722de0 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:25.627605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:25.629344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:25.712708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49336c02-9c08-4a7c-8d84-6243a489ae0d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:25.716176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d1926d34e824016927d5d2eeefca945', 'request_id': 'req-ecf1315b-d509-43e7-89fe-70bbb26dce7f'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.724352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d1926d34e824016927d5d2eeefca945', 'request_id': 'req-ecf1315b-d509-43e7-89fe-70bbb26dce7f'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.739441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] The token cache is being invalidate because project 9d1926d34e824016927d5d2eeefca945 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:25.739768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.740496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:25.742038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 842/1687] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:25 2022] DELETE /identity/v3/projects/9d1926d34e824016927d5d2eeefca945 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:25.746523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:25.747937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:25.810257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:25.811038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:25.811325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:25.811600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] PATH_INFO: `/v3/users/c57b5a9e2c324023ab46127d5827bf6b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:25.813148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] RBAC: Authorizing `identity:delete_user(user_id=c57b5a9e2c324023ab46127d5827bf6b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:25.815186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:25.816335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:25.838339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf1315b-d509-43e7-89fe-70bbb26dce7f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:25.845355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:25.845692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:25.845977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:25.846252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] PATH_INFO: `/v3/users/f125bfe20cbf4b97b2ecdbc1198daa5d` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:25.847520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f125bfe20cbf4b97b2ecdbc1198daa5d)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:25.849528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:26.064692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:26.067103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c57b5a9e2c324023ab46127d5827bf6b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.067769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c57b5a9e2c324023ab46127d5827bf6b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.093750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f125bfe20cbf4b97b2ecdbc1198daa5d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.096127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c57b5a9e2c324023ab46127d5827bf6b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.105855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c57b5a9e2c324023ab46127d5827bf6b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.109442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c57b5a9e2c324023ab46127d5827bf6b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.190685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f125bfe20cbf4b97b2ecdbc1198daa5d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.204880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f125bfe20cbf4b97b2ecdbc1198daa5d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.213095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f125bfe20cbf4b97b2ecdbc1198daa5d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.214109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f125bfe20cbf4b97b2ecdbc1198daa5d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:26.220791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 843/1688] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:25 2022] DELETE /identity/v3/users/f125bfe20cbf4b97b2ecdbc1198daa5d => generated 0 bytes in 475 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:26.266431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 846/1689] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:25 2022] DELETE /identity/v3/users/c57b5a9e2c324023ab46127d5827bf6b => generated 0 bytes in 639 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:26.809987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:26.812476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:26.930391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a75adde-d851-4304-b853-1008fbd92346 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:26.950984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:26.952499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:27.022661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.023397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.023673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.023950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] PATH_INFO: `/v3/projects/24ba12241be849b19b350c3755c6abd6` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.025089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] RBAC: Authorizing `identity:delete_project(project_id=24ba12241be849b19b350c3755c6abd6)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.027737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.040568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:27.053636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb1d945-4308-46b0-9abd-43b2254b3eff admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:27.059539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.060258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.060630 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.060979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] PATH_INFO: `/v3/projects/e2411fefbf584b849a3096e20d1351de` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.062173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2411fefbf584b849a3096e20d1351de)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.064193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.164986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24ba12241be849b19b350c3755c6abd6', 'request_id': 'req-f2d1531c-192c-4310-87e1-a92176e333da'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.225667 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2411fefbf584b849a3096e20d1351de', 'request_id': 'req-ccbb556d-39a7-4826-bbc4-125392f43f54'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.231964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2411fefbf584b849a3096e20d1351de', 'request_id': 'req-ccbb556d-39a7-4826-bbc4-125392f43f54'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.237129 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24ba12241be849b19b350c3755c6abd6', 'request_id': 'req-f2d1531c-192c-4310-87e1-a92176e333da'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.240494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2411fefbf584b849a3096e20d1351de', 'request_id': 'req-ccbb556d-39a7-4826-bbc4-125392f43f54'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.244333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24ba12241be849b19b350c3755c6abd6', 'request_id': 'req-f2d1531c-192c-4310-87e1-a92176e333da'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.256575 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] The token cache is being invalidate because project 24ba12241be849b19b350c3755c6abd6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:27.256860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.257526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f2d1531c-192c-4310-87e1-a92176e333da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.259017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 844/1690] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:26 2022] DELETE /identity/v3/projects/24ba12241be849b19b350c3755c6abd6 => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:27.264820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96de8571-b8c0-4c8a-956a-cd8c9c310be9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.265100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96de8571-b8c0-4c8a-956a-cd8c9c310be9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.265370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96de8571-b8c0-4c8a-956a-cd8c9c310be9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.284743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-96de8571-b8c0-4c8a-956a-cd8c9c310be9 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:27.285950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-96de8571-b8c0-4c8a-956a-cd8c9c310be9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:27.346007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 845/1691] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:27.353886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d44a265-aa08-4425-9253-0cc3e439af91 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.354171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d44a265-aa08-4425-9253-0cc3e439af91 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.354446 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d44a265-aa08-4425-9253-0cc3e439af91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.355960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] The token cache is being invalidate because project e2411fefbf584b849a3096e20d1351de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:27.356254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.362091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.362091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 847/1692] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:26 2022] DELETE /identity/v3/projects/e2411fefbf584b849a3096e20d1351de => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:27.364980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:27.366306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:27.377089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3d44a265-aa08-4425-9253-0cc3e439af91 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:27.378252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3d44a265-aa08-4425-9253-0cc3e439af91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:27.382452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbb556d-39a7-4826-bbc4-125392f43f54 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:27.387181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 846/1693] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:27.390700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.390700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.390700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.390700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] PATH_INFO: `/v3/users/b08dd7da0eba4ddba43301c23cbc9667` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.391075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b08dd7da0eba4ddba43301c23cbc9667)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.392237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.393872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d44a265-aa08-4425-9253-0cc3e439af91 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:27.403006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d44a265-aa08-4425-9253-0cc3e439af91 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:27.407700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.408298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.408580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.408854 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.409507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:27.409808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.411839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.427400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 847/1694] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:03:27.433326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:27.443065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dae1b08-4cb3-4a9d-b2df-76f17f2794fa admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:27.448276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdeebfd-623c-4eac-9a37-108bebcbfa66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.451332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdeebfd-623c-4eac-9a37-108bebcbfa66 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.451332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdeebfd-623c-4eac-9a37-108bebcbfa66 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.451332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdeebfd-623c-4eac-9a37-108bebcbfa66 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.451332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdeebfd-623c-4eac-9a37-108bebcbfa66 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.451903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdeebfd-623c-4eac-9a37-108bebcbfa66 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.561431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 848/1695] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:27 2022] POST /identity/v3/projects => generated 387 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:27.564900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdeebfd-623c-4eac-9a37-108bebcbfa66 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:27.578731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=18,keystone:SELECT=80,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:27.579957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b08dd7da0eba4ddba43301c23cbc9667'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.580732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b08dd7da0eba4ddba43301c23cbc9667'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.594191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b08dd7da0eba4ddba43301c23cbc9667'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.601580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b08dd7da0eba4ddba43301c23cbc9667'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.602526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b08dd7da0eba4ddba43301c23cbc9667'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:27.633587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdeebfd-623c-4eac-9a37-108bebcbfa66 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:27.639095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7b1b49-2747-45ff-b4b8-2113f1dd31fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.639453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b1b49-2747-45ff-b4b8-2113f1dd31fb admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.639731 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b1b49-2747-45ff-b4b8-2113f1dd31fb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.640000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7b1b49-2747-45ff-b4b8-2113f1dd31fb admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.640647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7b1b49-2747-45ff-b4b8-2113f1dd31fb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.642734 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7b1b49-2747-45ff-b4b8-2113f1dd31fb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.676266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 848/1696] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:27 2022] DELETE /identity/v3/users/b08dd7da0eba4ddba43301c23cbc9667 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:27.759655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 849/1697] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:27 2022] POST /identity/v3/users => generated 294 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:27.764075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:27.765636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:27.829749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf02a9-d7d3-4412-b434-54869ea8c991 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:27.834427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e53fb0-31af-4cdd-b2dc-fc09ebad3cb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.835046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e53fb0-31af-4cdd-b2dc-fc09ebad3cb0 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.835325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e53fb0-31af-4cdd-b2dc-fc09ebad3cb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.835593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e53fb0-31af-4cdd-b2dc-fc09ebad3cb0 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.836224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e53fb0-31af-4cdd-b2dc-fc09ebad3cb0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.837806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e53fb0-31af-4cdd-b2dc-fc09ebad3cb0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.847276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 849/1698] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:27.851471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7b1b49-2747-45ff-b4b8-2113f1dd31fb admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:27.861782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7b1b49-2747-45ff-b4b8-2113f1dd31fb admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:27.866289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b31c7a0-061b-454a-861b-b11bc2d9c547 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.866873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31c7a0-061b-454a-861b-b11bc2d9c547 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.867187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31c7a0-061b-454a-861b-b11bc2d9c547 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.867456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b31c7a0-061b-454a-861b-b11bc2d9c547 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.868095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b31c7a0-061b-454a-861b-b11bc2d9c547 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.870380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b31c7a0-061b-454a-861b-b11bc2d9c547 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.880473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 850/1699] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:27 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:27.886422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e53fb0-31af-4cdd-b2dc-fc09ebad3cb0 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:27.896167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e53fb0-31af-4cdd-b2dc-fc09ebad3cb0 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:27.901424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a03d05-17ef-4c6f-aa09-ef104f4d481e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:27.902069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a03d05-17ef-4c6f-aa09-ef104f4d481e admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:27.902346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a03d05-17ef-4c6f-aa09-ef104f4d481e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:27.902616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a03d05-17ef-4c6f-aa09-ef104f4d481e admin admin] PATH_INFO: `/v3/projects/611cd7b2006f4d278c2767ef44a0c657/users/7742f25b9fd6453f8bd7b4d48a4a33da/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:27.919254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a03d05-17ef-4c6f-aa09-ef104f4d481e admin admin] RBAC: Authorizing `identity:create_grant(project_id=611cd7b2006f4d278c2767ef44a0c657, user_id=7742f25b9fd6453f8bd7b4d48a4a33da, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:27.924793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a03d05-17ef-4c6f-aa09-ef104f4d481e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:27.945204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:28.009556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 850/1700] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:27 2022] PUT /identity/v3/projects/611cd7b2006f4d278c2767ef44a0c657/users/7742f25b9fd6453f8bd7b4d48a4a33da/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:28.013179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb6dd5-519e-4e2d-b2a7-e123247fcba5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.013725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb6dd5-519e-4e2d-b2a7-e123247fcba5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.014094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb6dd5-519e-4e2d-b2a7-e123247fcba5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.046204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-7fdb6dd5-519e-4e2d-b2a7-e123247fcba5 None None] MFA Rules not processed for user `7742f25b9fd6453f8bd7b4d48a4a33da`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:28.047410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7fdb6dd5-519e-4e2d-b2a7-e123247fcba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:28.128142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 851/1701] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:28.132959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a03d05-17ef-4c6f-aa09-ef104f4d481e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.158603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb6dd5-519e-4e2d-b2a7-e123247fcba5 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.160096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7fdb6dd5-519e-4e2d-b2a7-e123247fcba5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:28.219946 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a03d05-17ef-4c6f-aa09-ef104f4d481e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.224849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d28da69-a92a-416f-a2f3-f06dedd7b23b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.229403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d28da69-a92a-416f-a2f3-f06dedd7b23b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.229403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d28da69-a92a-416f-a2f3-f06dedd7b23b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.229403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d28da69-a92a-416f-a2f3-f06dedd7b23b admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.229403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d28da69-a92a-416f-a2f3-f06dedd7b23b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.230975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d28da69-a92a-416f-a2f3-f06dedd7b23b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.249110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb6dd5-519e-4e2d-b2a7-e123247fcba5 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.253871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.261462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.261462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.261462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] PATH_INFO: `/v3/projects/405ca0c3f13b4bbfaace1e39c454d101` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.261462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=405ca0c3f13b4bbfaace1e39c454d101)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.261462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.334855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 851/1702] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/projects => generated 387 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:28.339699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d28da69-a92a-416f-a2f3-f06dedd7b23b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.361554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=94,keystone:DELETE=3,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:28.379561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '405ca0c3f13b4bbfaace1e39c454d101', 'request_id': 'req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:28.403089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d28da69-a92a-416f-a2f3-f06dedd7b23b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.457285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.457897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.458174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.462146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.462146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.462146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.480800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '405ca0c3f13b4bbfaace1e39c454d101', 'request_id': 'req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:28.486807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '405ca0c3f13b4bbfaace1e39c454d101', 'request_id': 'req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:28.498706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] The token cache is being invalidate because project 405ca0c3f13b4bbfaace1e39c454d101 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:28.499092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:28.499767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-33e46be1-66c2-4352-ac1c-a37bfbbc68ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:28.501374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 852/1703] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:28 2022] DELETE /identity/v3/projects/405ca0c3f13b4bbfaace1e39c454d101 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:28.507092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a4a712-9787-42bd-95c0-fba667b33ce8 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.507380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a4a712-9787-42bd-95c0-fba667b33ce8 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.507657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a4a712-9787-42bd-95c0-fba667b33ce8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.528858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b5a4a712-9787-42bd-95c0-fba667b33ce8 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:28.530076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b5a4a712-9787-42bd-95c0-fba667b33ce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:28.538360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 853/1704] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:28.546508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f728b-e08c-4e7d-aeb9-ca6f553a6cb4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.546794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f728b-e08c-4e7d-aeb9-ca6f553a6cb4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.547112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537f728b-e08c-4e7d-aeb9-ca6f553a6cb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.566823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-537f728b-e08c-4e7d-aeb9-ca6f553a6cb4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:28.568085 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-537f728b-e08c-4e7d-aeb9-ca6f553a6cb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:28.573960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 852/1705] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/users => generated 294 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:28.577131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 854/1706] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:28.578166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.579874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:28.580834 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537f728b-e08c-4e7d-aeb9-ca6f553a6cb4 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.591622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537f728b-e08c-4e7d-aeb9-ca6f553a6cb4 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.592407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bff8571-6730-4286-aa55-b2b6410a70de admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.597008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a007c8c-998d-42cd-a8ab-0dcad583b7fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.601211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a007c8c-998d-42cd-a8ab-0dcad583b7fd admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.601211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a007c8c-998d-42cd-a8ab-0dcad583b7fd admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.601211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a007c8c-998d-42cd-a8ab-0dcad583b7fd admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.601211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a007c8c-998d-42cd-a8ab-0dcad583b7fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.601211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a007c8c-998d-42cd-a8ab-0dcad583b7fd admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.604805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.605480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.607175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=4,keystone:SELECT=89,keystone:DELETE=7 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:28.607722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.608011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.608789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:28.609100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.610837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.614121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 853/1707] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:03:28.618077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a007c8c-998d-42cd-a8ab-0dcad583b7fd admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.626477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 855/1708] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:03:28.627586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a007c8c-998d-42cd-a8ab-0dcad583b7fd admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.630605 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.632401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdda5bd3-3562-47f0-8537-6726b3b6b8b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.632992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdda5bd3-3562-47f0-8537-6726b3b6b8b4 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.633270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdda5bd3-3562-47f0-8537-6726b3b6b8b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.633540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdda5bd3-3562-47f0-8537-6726b3b6b8b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.634175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdda5bd3-3562-47f0-8537-6726b3b6b8b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.636433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdda5bd3-3562-47f0-8537-6726b3b6b8b4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.641981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca8c98f-4407-4d47-b677-15fc5dfdad53 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.646557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 854/1709] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:03:28.648318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae1428-5727-4633-8688-ea04eaece8ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.653368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae1428-5727-4633-8688-ea04eaece8ce admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.653368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae1428-5727-4633-8688-ea04eaece8ce admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.653368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae1428-5727-4633-8688-ea04eaece8ce admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.653368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae1428-5727-4633-8688-ea04eaece8ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.653368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaae1428-5727-4633-8688-ea04eaece8ce admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.654790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdda5bd3-3562-47f0-8537-6726b3b6b8b4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.665805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdda5bd3-3562-47f0-8537-6726b3b6b8b4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.671171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17891966-dac0-4551-a2d0-4d223bfd1073 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.671864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17891966-dac0-4551-a2d0-4d223bfd1073 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.672187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17891966-dac0-4551-a2d0-4d223bfd1073 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.672595 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17891966-dac0-4551-a2d0-4d223bfd1073 admin admin] PATH_INFO: `/v3/projects/e4ced483b0ce4203a36cfe8cdf2a0b22/users/708d7791f1e34cef8d6be36d9e4985b3/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.694921 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17891966-dac0-4551-a2d0-4d223bfd1073 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e4ced483b0ce4203a36cfe8cdf2a0b22, user_id=708d7791f1e34cef8d6be36d9e4985b3, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.698326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17891966-dac0-4551-a2d0-4d223bfd1073 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.734471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 856/1710] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/projects => generated 373 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:28.738402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae1428-5727-4633-8688-ea04eaece8ce admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.815569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 855/1711] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:28 2022] PUT /identity/v3/projects/e4ced483b0ce4203a36cfe8cdf2a0b22/users/708d7791f1e34cef8d6be36d9e4985b3/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:28.820262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaae1428-5727-4633-8688-ea04eaece8ce admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.830091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731e87f-1f66-4929-8287-0698b4176ac5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.830091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731e87f-1f66-4929-8287-0698b4176ac5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.830402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d731e87f-1f66-4929-8287-0698b4176ac5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.863222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d731e87f-1f66-4929-8287-0698b4176ac5 None None] MFA Rules not processed for user `708d7791f1e34cef8d6be36d9e4985b3`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:28.865394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d731e87f-1f66-4929-8287-0698b4176ac5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:28.904129 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cfd9e0-a4a4-4b14-9fd9-470107d182f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.904723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cfd9e0-a4a4-4b14-9fd9-470107d182f3 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.905002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cfd9e0-a4a4-4b14-9fd9-470107d182f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.905272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cfd9e0-a4a4-4b14-9fd9-470107d182f3 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.905918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cfd9e0-a4a4-4b14-9fd9-470107d182f3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.907877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cfd9e0-a4a4-4b14-9fd9-470107d182f3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:28.937495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:28.941572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 856/1712] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:28.945953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d731e87f-1f66-4929-8287-0698b4176ac5 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:28.956310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d731e87f-1f66-4929-8287-0698b4176ac5 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:28.960789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56d0b7-00a7-4286-8bd0-924d104a7b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:28.961357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56d0b7-00a7-4286-8bd0-924d104a7b5f admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:28.961632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56d0b7-00a7-4286-8bd0-924d104a7b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:28.961902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc56d0b7-00a7-4286-8bd0-924d104a7b5f admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:28.962514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc56d0b7-00a7-4286-8bd0-924d104a7b5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:28.964508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc56d0b7-00a7-4286-8bd0-924d104a7b5f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.036124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 857/1713] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/users => generated 287 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:29.039920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 857/1714] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:28 2022] POST /identity/v3/projects => generated 387 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:29.043288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cfd9e0-a4a4-4b14-9fd9-470107d182f3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.044305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56d0b7-00a7-4286-8bd0-924d104a7b5f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.053817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cfd9e0-a4a4-4b14-9fd9-470107d182f3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.101501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:29.113956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cd836e-643a-495c-ab98-92b088bb05e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.115394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cd836e-643a-495c-ab98-92b088bb05e9 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.115394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cd836e-643a-495c-ab98-92b088bb05e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.115394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cd836e-643a-495c-ab98-92b088bb05e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.115796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cd836e-643a-495c-ab98-92b088bb05e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.116834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc56d0b7-00a7-4286-8bd0-924d104a7b5f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.117354 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cd836e-643a-495c-ab98-92b088bb05e9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.122336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56e4b6f-b8f3-4061-8b06-e7ae18410896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.123384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4b6f-b8f3-4061-8b06-e7ae18410896 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.123384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4b6f-b8f3-4061-8b06-e7ae18410896 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.123615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4b6f-b8f3-4061-8b06-e7ae18410896 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.124159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56e4b6f-b8f3-4061-8b06-e7ae18410896 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.126040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56e4b6f-b8f3-4061-8b06-e7ae18410896 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.130922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 858/1715] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:03:29.132319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cd836e-643a-495c-ab98-92b088bb05e9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.142141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cd836e-643a-495c-ab98-92b088bb05e9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.146881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75471ff4-618a-45ed-9501-68066bbf0624 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.147515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75471ff4-618a-45ed-9501-68066bbf0624 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.147792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75471ff4-618a-45ed-9501-68066bbf0624 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.148064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75471ff4-618a-45ed-9501-68066bbf0624 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.148710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75471ff4-618a-45ed-9501-68066bbf0624 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.150310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75471ff4-618a-45ed-9501-68066bbf0624 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.164113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 859/1716] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:29.165160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75471ff4-618a-45ed-9501-68066bbf0624 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.175343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75471ff4-618a-45ed-9501-68066bbf0624 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.180411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b9994d-34df-4b20-9ed0-fefa2d5aba91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.181055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b9994d-34df-4b20-9ed0-fefa2d5aba91 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.181337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b9994d-34df-4b20-9ed0-fefa2d5aba91 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.181612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b9994d-34df-4b20-9ed0-fefa2d5aba91 admin admin] PATH_INFO: `/v3/projects/ec86749bbe5743d08c6b3c2ae8eade76/users/56b4866c6011456a8275cf84956373ca/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.198438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b9994d-34df-4b20-9ed0-fefa2d5aba91 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ec86749bbe5743d08c6b3c2ae8eade76, user_id=56b4866c6011456a8275cf84956373ca, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.201120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b9994d-34df-4b20-9ed0-fefa2d5aba91 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.253728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 858/1717] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:29 2022] POST /identity/v3/users => generated 300 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:29.257654 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56e4b6f-b8f3-4061-8b06-e7ae18410896 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.269408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56e4b6f-b8f3-4061-8b06-e7ae18410896 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.273614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 860/1718] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:29 2022] PUT /identity/v3/projects/ec86749bbe5743d08c6b3c2ae8eade76/users/56b4866c6011456a8275cf84956373ca/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:29.278627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a12802-cff7-43b1-9774-945836a90a07 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.279133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a12802-cff7-43b1-9774-945836a90a07 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.279435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a12802-cff7-43b1-9774-945836a90a07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.310079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-61a12802-cff7-43b1-9774-945836a90a07 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:29.311495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-61a12802-cff7-43b1-9774-945836a90a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:29.337622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca6f78a-9fb1-406d-80be-07c8bf3a171c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.338274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca6f78a-9fb1-406d-80be-07c8bf3a171c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.339123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca6f78a-9fb1-406d-80be-07c8bf3a171c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.339236 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca6f78a-9fb1-406d-80be-07c8bf3a171c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.339878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca6f78a-9fb1-406d-80be-07c8bf3a171c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.341761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca6f78a-9fb1-406d-80be-07c8bf3a171c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.361441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 859/1719] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:03:29.362365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca6f78a-9fb1-406d-80be-07c8bf3a171c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.390295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 861/1720] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:29.393549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca6f78a-9fb1-406d-80be-07c8bf3a171c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.400536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b36274-512f-4d90-85a2-a6710e05ae22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.401941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b36274-512f-4d90-85a2-a6710e05ae22 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.402266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b36274-512f-4d90-85a2-a6710e05ae22 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.402553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b36274-512f-4d90-85a2-a6710e05ae22 admin admin] PATH_INFO: `/v3/projects/04876561c63e428881c71c298cebf2d0/users/46f221e18e9a470bb9d2f0df9f592561/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.404248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a12802-cff7-43b1-9774-945836a90a07 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.406085 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-61a12802-cff7-43b1-9774-945836a90a07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:29.420347 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b36274-512f-4d90-85a2-a6710e05ae22 admin admin] RBAC: Authorizing `identity:create_grant(project_id=04876561c63e428881c71c298cebf2d0, user_id=46f221e18e9a470bb9d2f0df9f592561, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.423550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b36274-512f-4d90-85a2-a6710e05ae22 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.472035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a12802-cff7-43b1-9774-945836a90a07 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.475762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1701d9b6-013b-4ea7-a03c-47b314a3ef22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.476361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1701d9b6-013b-4ea7-a03c-47b314a3ef22 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.476642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1701d9b6-013b-4ea7-a03c-47b314a3ef22 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.476917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1701d9b6-013b-4ea7-a03c-47b314a3ef22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.486790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1701d9b6-013b-4ea7-a03c-47b314a3ef22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.488342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1701d9b6-013b-4ea7-a03c-47b314a3ef22 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.493450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 860/1721] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:29 2022] PUT /identity/v3/projects/04876561c63e428881c71c298cebf2d0/users/46f221e18e9a470bb9d2f0df9f592561/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:29.497465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b36274-512f-4d90-85a2-a6710e05ae22 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.499984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 862/1722] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:29.542168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:29.561313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b36274-512f-4d90-85a2-a6710e05ae22 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.565952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b512c1-c839-4128-b764-1e564241764e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.566557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b512c1-c839-4128-b764-1e564241764e admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.566840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b512c1-c839-4128-b764-1e564241764e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.567155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b512c1-c839-4128-b764-1e564241764e admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.567801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b512c1-c839-4128-b764-1e564241764e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.569397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b512c1-c839-4128-b764-1e564241764e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.579157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 861/1723] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:29 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:03:29.583399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1701d9b6-013b-4ea7-a03c-47b314a3ef22 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.593411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1701d9b6-013b-4ea7-a03c-47b314a3ef22 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.598221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.598872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.599191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.599466 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] PATH_INFO: `/v3/projects/04876561c63e428881c71c298cebf2d0/users/46f221e18e9a470bb9d2f0df9f592561/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.601418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] RBAC: Authorizing `identity:create_grant(project_id=04876561c63e428881c71c298cebf2d0, user_id=46f221e18e9a470bb9d2f0df9f592561, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.604685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.616182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 863/1724] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:29 2022] PUT /identity/v3/projects/04876561c63e428881c71c298cebf2d0/users/46f221e18e9a470bb9d2f0df9f592561/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:29.626931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920056e-cd8a-4219-b3f3-72eb87939c61 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.627460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920056e-cd8a-4219-b3f3-72eb87939c61 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.627744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5920056e-cd8a-4219-b3f3-72eb87939c61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.657585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5920056e-cd8a-4219-b3f3-72eb87939c61 None None] MFA Rules not processed for user `46f221e18e9a470bb9d2f0df9f592561`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:29.658820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5920056e-cd8a-4219-b3f3-72eb87939c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:29.727909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 862/1725] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:29 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:29.739376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:29.741337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:29.798911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:29.815086 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b56552-230b-4a5e-9fd4-60807554487b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:29.818933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eb7162-db0e-44ea-bfb2-a292d267b4e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:29.819592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eb7162-db0e-44ea-bfb2-a292d267b4e1 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:29.819871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eb7162-db0e-44ea-bfb2-a292d267b4e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:29.820149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eb7162-db0e-44ea-bfb2-a292d267b4e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:29.831133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75eb7162-db0e-44ea-bfb2-a292d267b4e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:29.832603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75eb7162-db0e-44ea-bfb2-a292d267b4e1 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:29.844140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 864/1726] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:29 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:31.033217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c92803-6e65-40fe-bfb7-4b12a63af062 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:31.033953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c92803-6e65-40fe-bfb7-4b12a63af062 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:31.034310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c92803-6e65-40fe-bfb7-4b12a63af062 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-82c92803-6e65-40fe-bfb7-4b12a63af062 None None] MFA Rules not processed for user `56b4866c6011456a8275cf84956373ca`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:31.082977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-82c92803-6e65-40fe-bfb7-4b12a63af062 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:31.095356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eb7162-db0e-44ea-bfb2-a292d267b4e1 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:31.106439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eb7162-db0e-44ea-bfb2-a292d267b4e1 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:31.114210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:31.115142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:31.115367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:31.115642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:31.117790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:31.174279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 863/1727] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:31 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:31.179728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c92803-6e65-40fe-bfb7-4b12a63af062 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:31.190053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:31.192060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:31.202933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 865/1728] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:31 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:31.239490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:31.251843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c92803-6e65-40fe-bfb7-4b12a63af062 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:31.257620 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68f9ad1-c405-4902-887b-7cea4c144f3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:31.258237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f9ad1-c405-4902-887b-7cea4c144f3b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:31.258518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f9ad1-c405-4902-887b-7cea4c144f3b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:31.258790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68f9ad1-c405-4902-887b-7cea4c144f3b admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:31.259468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68f9ad1-c405-4902-887b-7cea4c144f3b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:31.261947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68f9ad1-c405-4902-887b-7cea4c144f3b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:31.390266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 864/1729] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:31 2022] POST /identity/v3/projects => generated 373 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:31.391528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:31.403197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd5561a-70e9-4be9-b583-6db8b6f0347e service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:31.472368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d94415-672a-4374-84d3-19eed4159bf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:31.472979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d94415-672a-4374-84d3-19eed4159bf9 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:31.473305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d94415-672a-4374-84d3-19eed4159bf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:31.473531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d94415-672a-4374-84d3-19eed4159bf9 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:31.474177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d94415-672a-4374-84d3-19eed4159bf9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:31.476249 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d94415-672a-4374-84d3-19eed4159bf9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:31.574630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 866/1730] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:31 2022] POST /identity/v3/users => generated 293 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:31.578545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68f9ad1-c405-4902-887b-7cea4c144f3b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:31.597723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68f9ad1-c405-4902-887b-7cea4c144f3b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:31.602352 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd946b49-28bd-4a13-b80b-41885c430330 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:31.602979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd946b49-28bd-4a13-b80b-41885c430330 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:31.603253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd946b49-28bd-4a13-b80b-41885c430330 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:31.603528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd946b49-28bd-4a13-b80b-41885c430330 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:31.604223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd946b49-28bd-4a13-b80b-41885c430330 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:31.605843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd946b49-28bd-4a13-b80b-41885c430330 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:31.615724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 865/1731] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:31 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:31.619946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d94415-672a-4374-84d3-19eed4159bf9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:31.630081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d94415-672a-4374-84d3-19eed4159bf9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:31.634684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15145c08-d4b2-4f26-972e-9399c38539ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:31.635883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15145c08-d4b2-4f26-972e-9399c38539ea admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:31.636176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15145c08-d4b2-4f26-972e-9399c38539ea admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:31.636481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15145c08-d4b2-4f26-972e-9399c38539ea admin admin] PATH_INFO: `/v3/projects/d4b4cb4efb26452496accb58164b8f21/users/5be43f8aa983469e8fa7ff4352004582/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:31.652580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15145c08-d4b2-4f26-972e-9399c38539ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4b4cb4efb26452496accb58164b8f21, user_id=5be43f8aa983469e8fa7ff4352004582, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:31.655417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15145c08-d4b2-4f26-972e-9399c38539ea admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:31.715704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 867/1732] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:31 2022] PUT /identity/v3/projects/d4b4cb4efb26452496accb58164b8f21/users/5be43f8aa983469e8fa7ff4352004582/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:31.719785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd946b49-28bd-4a13-b80b-41885c430330 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:31.729970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd946b49-28bd-4a13-b80b-41885c430330 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:31.802469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d06e4a6-2060-4fb9-8bbe-63b9dfaa5d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:31.803433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d06e4a6-2060-4fb9-8bbe-63b9dfaa5d5c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:31.803714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d06e4a6-2060-4fb9-8bbe-63b9dfaa5d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:31.803989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d06e4a6-2060-4fb9-8bbe-63b9dfaa5d5c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:31.804683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d06e4a6-2060-4fb9-8bbe-63b9dfaa5d5c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:31.806528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d06e4a6-2060-4fb9-8bbe-63b9dfaa5d5c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:31.817797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 866/1733] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:31 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:31.823826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15145c08-d4b2-4f26-972e-9399c38539ea admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:31.851041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15145c08-d4b2-4f26-972e-9399c38539ea admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:31.861418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:31.862204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:31.862488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:31.862776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] PATH_INFO: `/v3/projects/d4b4cb4efb26452496accb58164b8f21/users/5be43f8aa983469e8fa7ff4352004582/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:31.864772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4b4cb4efb26452496accb58164b8f21, user_id=5be43f8aa983469e8fa7ff4352004582, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:31.867485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:31.882719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 868/1734] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:31 2022] PUT /identity/v3/projects/d4b4cb4efb26452496accb58164b8f21/users/5be43f8aa983469e8fa7ff4352004582/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:33.743081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c5fe06-1e05-4946-a830-accab378110c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:33.743987 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c5fe06-1e05-4946-a830-accab378110c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:33.744476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c5fe06-1e05-4946-a830-accab378110c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:33.830597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f9c5fe06-1e05-4946-a830-accab378110c None None] MFA Rules not processed for user `5be43f8aa983469e8fa7ff4352004582`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:33.832373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f9c5fe06-1e05-4946-a830-accab378110c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:33.952972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 867/1735] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:33 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:33.965582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:33.967477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:34.019138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:34.045092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5cd643-00e5-49e0-8323-e3fd3d524cd6 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:34.051110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bb15b4-6db2-47b5-ac11-9643d9ff779b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:34.051902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bb15b4-6db2-47b5-ac11-9643d9ff779b service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:34.052181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bb15b4-6db2-47b5-ac11-9643d9ff779b service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:34.052455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bb15b4-6db2-47b5-ac11-9643d9ff779b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:34.127289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bb15b4-6db2-47b5-ac11-9643d9ff779b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:34.128902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bb15b4-6db2-47b5-ac11-9643d9ff779b service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:34.141735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 869/1736] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:33 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:36.845190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e5ad2e-6a9d-48e2-aa41-7a0da4d27b65 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:36.845648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e5ad2e-6a9d-48e2-aa41-7a0da4d27b65 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:36.845934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e5ad2e-6a9d-48e2-aa41-7a0da4d27b65 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:36.847495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 868/1737] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:03:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:03:36.858259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339ad65-cc6a-4673-99dd-248e5d51b744 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:36.858786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339ad65-cc6a-4673-99dd-248e5d51b744 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:36.858899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e339ad65-cc6a-4673-99dd-248e5d51b744 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:36.879201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e339ad65-cc6a-4673-99dd-248e5d51b744 None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:36.880402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e339ad65-cc6a-4673-99dd-248e5d51b744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:36.949520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 870/1738] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:03:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:36.960323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e5ad2e-6a9d-48e2-aa41-7a0da4d27b65 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:36.962121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a1e5ad2e-6a9d-48e2-aa41-7a0da4d27b65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:36.978060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e5ad2e-6a9d-48e2-aa41-7a0da4d27b65 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:36.983101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ab7138-1a78-46d4-9b4e-897e50d357ae service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:36.983776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ab7138-1a78-46d4-9b4e-897e50d357ae service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:36.984115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ab7138-1a78-46d4-9b4e-897e50d357ae service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:36.984453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ab7138-1a78-46d4-9b4e-897e50d357ae service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:36.993252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ab7138-1a78-46d4-9b4e-897e50d357ae service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:36.995595 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ab7138-1a78-46d4-9b4e-897e50d357ae service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:37.005938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 869/1739] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:36 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:37.043774 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339ad65-cc6a-4673-99dd-248e5d51b744 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:37.045230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e339ad65-cc6a-4673-99dd-248e5d51b744 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:37.112431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e339ad65-cc6a-4673-99dd-248e5d51b744 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:37.117078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:37.117715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:37.117997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:37.118280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] PATH_INFO: `/v3/users/53639656d3a542c0b822ea8084b2e2d5` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:37.120004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] RBAC: Authorizing `identity:delete_user(user_id=53639656d3a542c0b822ea8084b2e2d5)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:37.121934 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:37.144610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:37.305494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53639656d3a542c0b822ea8084b2e2d5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:37.454409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53639656d3a542c0b822ea8084b2e2d5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:37.470450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53639656d3a542c0b822ea8084b2e2d5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:37.478179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53639656d3a542c0b822ea8084b2e2d5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:37.479677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d50498a-b475-48fd-a662-3a8ee4b07e36 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53639656d3a542c0b822ea8084b2e2d5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:37.489920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 871/1740] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:37 2022] DELETE /identity/v3/users/53639656d3a542c0b822ea8084b2e2d5 => generated 0 bytes in 446 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:37.772937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ab7138-1a78-46d4-9b4e-897e50d357ae service placement] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:37.789519 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ab7138-1a78-46d4-9b4e-897e50d357ae service placement] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:37.822865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:37.849146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:37.849804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:37.850082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:37.850363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] PATH_INFO: `/v3/projects/95e2ff128466409e98277d6e1c620fca` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:37.851535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95e2ff128466409e98277d6e1c620fca)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:37.854149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:37.983600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95e2ff128466409e98277d6e1c620fca', 'request_id': 'req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:37.989935 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95e2ff128466409e98277d6e1c620fca', 'request_id': 'req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:37.995916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95e2ff128466409e98277d6e1c620fca', 'request_id': 'req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:38.093424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] The token cache is being invalidate because project 95e2ff128466409e98277d6e1c620fca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:38.093717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:38.094395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-27ab10a3-f87f-4307-8fcc-0b688b1e8b12 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:38.096110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 870/1741] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:37 2022] DELETE /identity/v3/projects/95e2ff128466409e98277d6e1c620fca => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:38.100927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3aec2-083d-49dc-ab7f-893a410f68bc None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.101442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3aec2-083d-49dc-ab7f-893a410f68bc None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.101539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23c3aec2-083d-49dc-ab7f-893a410f68bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.122580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-23c3aec2-083d-49dc-ab7f-893a410f68bc None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:38.123834 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-23c3aec2-083d-49dc-ab7f-893a410f68bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:38.183133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 872/1742] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:38.190680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56acd3fc-55a1-40cb-9dca-f9358a4968d0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.190979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56acd3fc-55a1-40cb-9dca-f9358a4968d0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.191266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56acd3fc-55a1-40cb-9dca-f9358a4968d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.211276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-56acd3fc-55a1-40cb-9dca-f9358a4968d0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:38.212465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-56acd3fc-55a1-40cb-9dca-f9358a4968d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:38.220883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 871/1743] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:38.224959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c3aec2-083d-49dc-ab7f-893a410f68bc None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:38.234495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23c3aec2-083d-49dc-ab7f-893a410f68bc None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:38.239063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:38.239650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.239924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.240192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.240836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:38.241162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:38.242751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:38.254176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 873/1744] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:03:38.258304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56acd3fc-55a1-40cb-9dca-f9358a4968d0 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:38.267854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56acd3fc-55a1-40cb-9dca-f9358a4968d0 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:38.272838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bcc741-8259-4f29-837f-6e261e4863f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:38.273434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bcc741-8259-4f29-837f-6e261e4863f1 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.273707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bcc741-8259-4f29-837f-6e261e4863f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.285054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bcc741-8259-4f29-837f-6e261e4863f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.285054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bcc741-8259-4f29-837f-6e261e4863f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:38.285054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bcc741-8259-4f29-837f-6e261e4863f1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:38.383207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 872/1745] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:38 2022] POST /identity/v3/projects => generated 377 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:38.384472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:38.395355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e0081-39ba-4768-83c8-8f594fe4e7a2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:38.452306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5288816e-cc8e-4bb7-9fa2-61f553682d89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:38.452910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5288816e-cc8e-4bb7-9fa2-61f553682d89 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.453193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5288816e-cc8e-4bb7-9fa2-61f553682d89 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.453470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5288816e-cc8e-4bb7-9fa2-61f553682d89 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.454102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5288816e-cc8e-4bb7-9fa2-61f553682d89 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:38.456052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5288816e-cc8e-4bb7-9fa2-61f553682d89 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:38.623312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 874/1746] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:38 2022] POST /identity/v3/users => generated 289 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:38.623424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bcc741-8259-4f29-837f-6e261e4863f1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:38.630782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bcc741-8259-4f29-837f-6e261e4863f1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:38.636804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:38.640348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.640348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.640348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.640348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:38.640348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:38.653751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 873/1747] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:03:38.654928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5288816e-cc8e-4bb7-9fa2-61f553682d89 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:38.675130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5288816e-cc8e-4bb7-9fa2-61f553682d89 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:38.683238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:38.683894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51200596-e37e-422b-a6f7-2e0c60d341a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:38.684585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51200596-e37e-422b-a6f7-2e0c60d341a3 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.684932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51200596-e37e-422b-a6f7-2e0c60d341a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.685232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:38.685364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51200596-e37e-422b-a6f7-2e0c60d341a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.686589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51200596-e37e-422b-a6f7-2e0c60d341a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:38.688597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51200596-e37e-422b-a6f7-2e0c60d341a3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:38.705589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 875/1748] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:38 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:38.707300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51200596-e37e-422b-a6f7-2e0c60d341a3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:38.722623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51200596-e37e-422b-a6f7-2e0c60d341a3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:38.733747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40bdbd33-8e29-498b-ace7-f29404ecff64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:38.734652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bdbd33-8e29-498b-ace7-f29404ecff64 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.735038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bdbd33-8e29-498b-ace7-f29404ecff64 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.735349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bdbd33-8e29-498b-ace7-f29404ecff64 admin admin] PATH_INFO: `/v3/projects/2caa701e503b44ddb9fbe09d36a53853/users/5ae13baf5df84b588fa65e8dffec4bbf/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.738535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:38.752553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bdbd33-8e29-498b-ace7-f29404ecff64 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2caa701e503b44ddb9fbe09d36a53853, user_id=5ae13baf5df84b588fa65e8dffec4bbf, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:38.755271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bdbd33-8e29-498b-ace7-f29404ecff64 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:38.755601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e793f30-5266-4016-b6ef-cacdd64bf0c5 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:38.759388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:38.760636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.760852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.761121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.763016 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:38.808270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:38.824405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 876/1749] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:38 2022] PUT /identity/v3/projects/2caa701e503b44ddb9fbe09d36a53853/users/5ae13baf5df84b588fa65e8dffec4bbf/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:38.828875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49290d66-ed51-45dc-be3b-ce984b42ca16 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:38.829331 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49290d66-ed51-45dc-be3b-ce984b42ca16 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:38.829738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49290d66-ed51-45dc-be3b-ce984b42ca16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:38.839137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:38.840752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:38.859142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 874/1750] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:38 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:38.876266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-49290d66-ed51-45dc-be3b-ce984b42ca16 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:38.878827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-49290d66-ed51-45dc-be3b-ce984b42ca16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:38.967610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 877/1751] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:38.980232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:38.995082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:39.066920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d086ce-f3ab-4215-ac3d-5520366632c8 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:39.120597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:39.121219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:39.121543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:39.121820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:39.131055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:39.132478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:39.142639 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 875/1752] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:39.165750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49290d66-ed51-45dc-be3b-ce984b42ca16 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:39.167338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-49290d66-ed51-45dc-be3b-ce984b42ca16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:39.227157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49290d66-ed51-45dc-be3b-ce984b42ca16 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:39.231630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f512d2-6bf8-4eab-8bae-a665a764afe4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:39.232415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f512d2-6bf8-4eab-8bae-a665a764afe4 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:39.232537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f512d2-6bf8-4eab-8bae-a665a764afe4 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:39.232788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f512d2-6bf8-4eab-8bae-a665a764afe4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:39.234700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-94f512d2-6bf8-4eab-8bae-a665a764afe4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:39.308959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f512d2-6bf8-4eab-8bae-a665a764afe4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:39.311226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f512d2-6bf8-4eab-8bae-a665a764afe4 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:39.334209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 878/1753] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:39 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:44.242060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:44.243583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:44.262857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef87ecc1-bfa8-4eb4-8dc8-48596fde95a7 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:44.268219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:44.271102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:44.271584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:44.272078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] PATH_INFO: `/v3/users/7742f25b9fd6453f8bd7b4d48a4a33da` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:44.273494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7742f25b9fd6453f8bd7b4d48a4a33da)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:44.275952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:44.592490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7742f25b9fd6453f8bd7b4d48a4a33da'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:44.593898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7742f25b9fd6453f8bd7b4d48a4a33da'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:44.614592 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7742f25b9fd6453f8bd7b4d48a4a33da'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:44.627156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7742f25b9fd6453f8bd7b4d48a4a33da'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:44.628591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7742f25b9fd6453f8bd7b4d48a4a33da'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:44.708546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 876/1754] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:44 2022] DELETE /identity/v3/users/7742f25b9fd6453f8bd7b4d48a4a33da => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:44.713712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1a1b8-9f51-4a69-940b-d19e69994e77 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:44.714498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1a1b8-9f51-4a69-940b-d19e69994e77 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:44.714779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f1a1b8-9f51-4a69-940b-d19e69994e77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:44.741068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a6f1a1b8-9f51-4a69-940b-d19e69994e77 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:44.742373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a6f1a1b8-9f51-4a69-940b-d19e69994e77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:44.799151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:44.843171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 879/1755] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:44.854927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:44.856416 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1a1b8-9f51-4a69-940b-d19e69994e77 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:44.862940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a6f1a1b8-9f51-4a69-940b-d19e69994e77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:44.866355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:44.875464 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:44.888114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f1a1b8-9f51-4a69-940b-d19e69994e77 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:44.895396 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:44.896546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:44.897008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:44.897429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] PATH_INFO: `/v3/users/e4c00ad1a2544535adc54e60f7749e99` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:44.898784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4c00ad1a2544535adc54e60f7749e99)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:44.901584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:44.948339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da2d9b-a12f-4d1a-974b-c645820cf096 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:44.952718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b80cf21-9499-4992-b744-1cd809c3268d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:44.953487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80cf21-9499-4992-b744-1cd809c3268d service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:44.953830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80cf21-9499-4992-b744-1cd809c3268d service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:44.953954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80cf21-9499-4992-b744-1cd809c3268d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:44.968030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b80cf21-9499-4992-b744-1cd809c3268d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:44.969490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b80cf21-9499-4992-b744-1cd809c3268d service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:44.984710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 877/1756] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:45.184828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4c00ad1a2544535adc54e60f7749e99'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:45.346723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b41650-0b98-4b1a-bacc-f79f7650ff52 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:45.347473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b41650-0b98-4b1a-bacc-f79f7650ff52 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:45.347473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b41650-0b98-4b1a-bacc-f79f7650ff52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:45.384370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b9b41650-0b98-4b1a-bacc-f79f7650ff52 None None] MFA Rules not processed for user `5ae13baf5df84b588fa65e8dffec4bbf`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:45.386559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b9b41650-0b98-4b1a-bacc-f79f7650ff52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:45.396017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4c00ad1a2544535adc54e60f7749e99'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:45.421281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4c00ad1a2544535adc54e60f7749e99'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:45.449213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4c00ad1a2544535adc54e60f7749e99'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:45.449213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-540932ad-b8c5-430e-b320-bf12b93457c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4c00ad1a2544535adc54e60f7749e99'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:45.466230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 880/1757] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:44 2022] DELETE /identity/v3/users/e4c00ad1a2544535adc54e60f7749e99 => generated 0 bytes in 610 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:45.473894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905449de-218a-43ec-ad5e-1802e8db1555 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:45.474243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905449de-218a-43ec-ad5e-1802e8db1555 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:45.474582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905449de-218a-43ec-ad5e-1802e8db1555 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:45.552520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-905449de-218a-43ec-ad5e-1802e8db1555 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:45.553356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 878/1758] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:45.553598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-905449de-218a-43ec-ad5e-1802e8db1555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:45.572694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b41650-0b98-4b1a-bacc-f79f7650ff52 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:45.574360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b9b41650-0b98-4b1a-bacc-f79f7650ff52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:45.673178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 881/1759] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:45.693365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905449de-218a-43ec-ad5e-1802e8db1555 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:45.694966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-905449de-218a-43ec-ad5e-1802e8db1555 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:45.747386 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b41650-0b98-4b1a-bacc-f79f7650ff52 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:45.809824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:45.815589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905449de-218a-43ec-ad5e-1802e8db1555 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:45.820510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:45.822108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:45.822674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:45.823275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:45.831502 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:45.837842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:45.842148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:45.853830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 882/1760] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:45.856416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b4cd0-377a-4d9c-a505-07268a193b0c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:45.857051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b4cd0-377a-4d9c-a505-07268a193b0c service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:45.857329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b4cd0-377a-4d9c-a505-07268a193b0c service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:45.857603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70b4cd0-377a-4d9c-a505-07268a193b0c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:45.902375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:45.905285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:45.933970 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f801c6-ab71-46c4-9cdf-1316bcf42a9c service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:45.936341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70b4cd0-377a-4d9c-a505-07268a193b0c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:45.943770 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a70b4cd0-377a-4d9c-a505-07268a193b0c service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:45.944710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:45.946499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:45.946829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:45.947397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] PATH_INFO: `/v3/projects/611cd7b2006f4d278c2767ef44a0c657` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:45.948895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] RBAC: Authorizing `identity:delete_project(project_id=611cd7b2006f4d278c2767ef44a0c657)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:45.951076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:45.956925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 879/1761] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 385 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:46.112329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '611cd7b2006f4d278c2767ef44a0c657', 'request_id': 'req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.217927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '611cd7b2006f4d278c2767ef44a0c657', 'request_id': 'req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.233709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '611cd7b2006f4d278c2767ef44a0c657', 'request_id': 'req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.254317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] The token cache is being invalidate because project 611cd7b2006f4d278c2767ef44a0c657 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:46.254925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.255829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.258012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 883/1762] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:45 2022] DELETE /identity/v3/projects/611cd7b2006f4d278c2767ef44a0c657 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:46.264130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b4cd0-377a-4d9c-a505-07268a193b0c service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:46.279069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a70b4cd0-377a-4d9c-a505-07268a193b0c service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:46.361791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:46.362499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:46.362714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:46.363022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] PATH_INFO: `/v3/users/708d7791f1e34cef8d6be36d9e4985b3` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:46.364178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=708d7791f1e34cef8d6be36d9e4985b3)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:46.366301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:46.420098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:46.421598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:46.434978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8b5e54-7134-44f9-8ab6-6d3e0a7ea99e admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:46.440094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:46.446196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:46.446196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:46.446196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] PATH_INFO: `/v3/projects/31156c69a6ea49799ae059f8e611da44` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:46.446196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] RBAC: Authorizing `identity:delete_project(project_id=31156c69a6ea49799ae059f8e611da44)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:46.446196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:46.639190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '708d7791f1e34cef8d6be36d9e4985b3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.640043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '708d7791f1e34cef8d6be36d9e4985b3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.641598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31156c69a6ea49799ae059f8e611da44', 'request_id': 'req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.655881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '708d7791f1e34cef8d6be36d9e4985b3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.663047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '708d7791f1e34cef8d6be36d9e4985b3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.664074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '708d7791f1e34cef8d6be36d9e4985b3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.665858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=91,keystone:DELETE=9 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:46.755781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31156c69a6ea49799ae059f8e611da44', 'request_id': 'req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.765785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31156c69a6ea49799ae059f8e611da44', 'request_id': 'req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.779402 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] The token cache is being invalidate because project 31156c69a6ea49799ae059f8e611da44 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:46.779703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.780384 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:46.782174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 884/1763] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:46 2022] DELETE /identity/v3/projects/31156c69a6ea49799ae059f8e611da44 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:46.887831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 880/1764] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:46 2022] DELETE /identity/v3/users/708d7791f1e34cef8d6be36d9e4985b3 => generated 0 bytes in 625 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:47.422772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:47.424533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:47.545897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d8e8e7-0273-4d6a-a60f-2af0c953a687 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:47.552108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:47.625301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:47.626005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:47.626293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:47.626570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] PATH_INFO: `/v3/projects/e4ced483b0ce4203a36cfe8cdf2a0b22` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:47.627762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] RBAC: Authorizing `identity:delete_project(project_id=e4ced483b0ce4203a36cfe8cdf2a0b22)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:47.629772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:47.751285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e4ced483b0ce4203a36cfe8cdf2a0b22', 'request_id': 'req-cc660138-572d-4e50-a316-40281aab827d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:47.825635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e4ced483b0ce4203a36cfe8cdf2a0b22', 'request_id': 'req-cc660138-572d-4e50-a316-40281aab827d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:47.831867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e4ced483b0ce4203a36cfe8cdf2a0b22', 'request_id': 'req-cc660138-572d-4e50-a316-40281aab827d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:47.845649 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] The token cache is being invalidate because project e4ced483b0ce4203a36cfe8cdf2a0b22 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:47.845927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:47.846816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cc660138-572d-4e50-a316-40281aab827d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:47.848560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 885/1765] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:47 2022] DELETE /identity/v3/projects/e4ced483b0ce4203a36cfe8cdf2a0b22 => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:47.853483 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:47.856855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:47.871117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de96ccfa-7bfb-4aad-a8b7-6af96a8356cd admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:47.950576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:47.951482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:47.951927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:47.952369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] PATH_INFO: `/v3/users/46f221e18e9a470bb9d2f0df9f592561` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:47.953679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=46f221e18e9a470bb9d2f0df9f592561)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:47.956254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:48.176981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '46f221e18e9a470bb9d2f0df9f592561'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:48.177763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '46f221e18e9a470bb9d2f0df9f592561'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:48.199812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '46f221e18e9a470bb9d2f0df9f592561'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:48.207376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '46f221e18e9a470bb9d2f0df9f592561'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:48.208312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '46f221e18e9a470bb9d2f0df9f592561'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:48.351851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 881/1766] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:47 2022] DELETE /identity/v3/users/46f221e18e9a470bb9d2f0df9f592561 => generated 0 bytes in 496 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:48.781555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79152157-4978-4c4f-b173-536bb2e19a06 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:48.781996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79152157-4978-4c4f-b173-536bb2e19a06 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:48.782114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79152157-4978-4c4f-b173-536bb2e19a06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:48.818314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-79152157-4978-4c4f-b173-536bb2e19a06 None None] MFA Rules not processed for user `5ae13baf5df84b588fa65e8dffec4bbf`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:48.819085 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-79152157-4978-4c4f-b173-536bb2e19a06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:48.886343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 886/1767] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:48 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:48.945477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:48.947934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:49.044856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-958169ac-646c-41ed-b1b9-2224cd105f1e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:49.050115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:49.052851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:49.052851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:49.052851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] PATH_INFO: `/v3/projects/04876561c63e428881c71c298cebf2d0` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:49.052851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=04876561c63e428881c71c298cebf2d0)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:49.055036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:49.068118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:INSERT=2,keystone:DELETE=11,keystone:SELECT=87 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:49.175854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04876561c63e428881c71c298cebf2d0', 'request_id': 'req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:49.183430 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04876561c63e428881c71c298cebf2d0', 'request_id': 'req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:49.190020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04876561c63e428881c71c298cebf2d0', 'request_id': 'req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:49.348026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] The token cache is being invalidate because project 04876561c63e428881c71c298cebf2d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:49.348774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:49.355638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3732378b-b9be-4cd8-9fbd-bb99e80c26f3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:49.359449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 882/1768] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:48 2022] DELETE /identity/v3/projects/04876561c63e428881c71c298cebf2d0 => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:49.365395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8fb82e-2ef1-4521-b163-12461ef8bc90 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:49.365680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8fb82e-2ef1-4521-b163-12461ef8bc90 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:49.365953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e8fb82e-2ef1-4521-b163-12461ef8bc90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:49.394939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1e8fb82e-2ef1-4521-b163-12461ef8bc90 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:49.397401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1e8fb82e-2ef1-4521-b163-12461ef8bc90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:49.505442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:49.541805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 887/1769] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:49.551395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fed757-7f13-4f75-8a81-2c2952186967 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:49.551935 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fed757-7f13-4f75-8a81-2c2952186967 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:49.556330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53fed757-7f13-4f75-8a81-2c2952186967 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:49.590137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-53fed757-7f13-4f75-8a81-2c2952186967 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:49.591534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-53fed757-7f13-4f75-8a81-2c2952186967 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:49.610627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 883/1770] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:49.614474 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8fb82e-2ef1-4521-b163-12461ef8bc90 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:49.635938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e8fb82e-2ef1-4521-b163-12461ef8bc90 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:49.642853 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:49.643551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:49.643867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:49.644146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:49.644812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:49.645111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:49.646728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:49.672317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 888/1771] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:03:49.673371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fed757-7f13-4f75-8a81-2c2952186967 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:49.686794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53fed757-7f13-4f75-8a81-2c2952186967 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:49.695578 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b73ba7-52d7-4142-9335-f70094b7896c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:49.696496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b73ba7-52d7-4142-9335-f70094b7896c admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:49.696828 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b73ba7-52d7-4142-9335-f70094b7896c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:49.697624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b73ba7-52d7-4142-9335-f70094b7896c admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:49.698786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b73ba7-52d7-4142-9335-f70094b7896c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:49.707650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b73ba7-52d7-4142-9335-f70094b7896c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:49.844756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 884/1772] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:49 2022] POST /identity/v3/projects => generated 399 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:49.849056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:49.861110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04ae2ec-20bc-4f2f-9a03-51aef1eb7be9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:49.940560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56cdbf4-02c3-45c8-825b-e763e7875278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:49.941422 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56cdbf4-02c3-45c8-825b-e763e7875278 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:49.941796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56cdbf4-02c3-45c8-825b-e763e7875278 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:49.942124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f56cdbf4-02c3-45c8-825b-e763e7875278 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:49.942787 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56cdbf4-02c3-45c8-825b-e763e7875278 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:49.944802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f56cdbf4-02c3-45c8-825b-e763e7875278 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:50.041988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 889/1773] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:49 2022] POST /identity/v3/users => generated 300 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:50.046109 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b73ba7-52d7-4142-9335-f70094b7896c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:50.057356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b73ba7-52d7-4142-9335-f70094b7896c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:50.061831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3845f0-3d43-49a3-9209-0b02a91df6a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:50.062461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3845f0-3d43-49a3-9209-0b02a91df6a5 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:50.062751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3845f0-3d43-49a3-9209-0b02a91df6a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:50.063053 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3845f0-3d43-49a3-9209-0b02a91df6a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:50.063693 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3845f0-3d43-49a3-9209-0b02a91df6a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:50.065292 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3845f0-3d43-49a3-9209-0b02a91df6a5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:50.075104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 885/1774] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:50.082974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56cdbf4-02c3-45c8-825b-e763e7875278 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:50.093400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f56cdbf4-02c3-45c8-825b-e763e7875278 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:50.110119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f09e15-a336-4dc9-a381-b2522039f42b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:50.110841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f09e15-a336-4dc9-a381-b2522039f42b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:50.111267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f09e15-a336-4dc9-a381-b2522039f42b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:50.111449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f09e15-a336-4dc9-a381-b2522039f42b admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:50.112109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f09e15-a336-4dc9-a381-b2522039f42b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:50.113868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f09e15-a336-4dc9-a381-b2522039f42b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:50.138052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 890/1775] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:50 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:50.141196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3845f0-3d43-49a3-9209-0b02a91df6a5 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:50.153994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3845f0-3d43-49a3-9209-0b02a91df6a5 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:50.162418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30c4525-667e-42bb-933b-0ef96fc1c310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:50.163415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30c4525-667e-42bb-933b-0ef96fc1c310 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:50.166443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30c4525-667e-42bb-933b-0ef96fc1c310 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:50.166443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30c4525-667e-42bb-933b-0ef96fc1c310 admin admin] PATH_INFO: `/v3/projects/9ed921dbf8b14e46a488ba4b7f412087/users/1a464fd6d93641f1b1c4f3d7bbf29b91/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:50.184119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30c4525-667e-42bb-933b-0ef96fc1c310 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ed921dbf8b14e46a488ba4b7f412087, user_id=1a464fd6d93641f1b1c4f3d7bbf29b91, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:50.187142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30c4525-667e-42bb-933b-0ef96fc1c310 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:50.266550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 886/1776] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:50 2022] PUT /identity/v3/projects/9ed921dbf8b14e46a488ba4b7f412087/users/1a464fd6d93641f1b1c4f3d7bbf29b91/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:50.281176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9bc3c-88a7-41b4-8e1a-0e4625f86550 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:50.281469 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9bc3c-88a7-41b4-8e1a-0e4625f86550 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:50.281727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e9bc3c-88a7-41b4-8e1a-0e4625f86550 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:50.313937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-96e9bc3c-88a7-41b4-8e1a-0e4625f86550 None None] MFA Rules not processed for user `1a464fd6d93641f1b1c4f3d7bbf29b91`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:50.315456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-96e9bc3c-88a7-41b4-8e1a-0e4625f86550 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:50.382274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 891/1777] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:50 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:50.386806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30c4525-667e-42bb-933b-0ef96fc1c310 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:50.448753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30c4525-667e-42bb-933b-0ef96fc1c310 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:50.453534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574291ec-82e3-47ff-a6a7-b451b631146b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:50.454134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574291ec-82e3-47ff-a6a7-b451b631146b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:50.454421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574291ec-82e3-47ff-a6a7-b451b631146b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:50.454696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574291ec-82e3-47ff-a6a7-b451b631146b admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:50.455365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574291ec-82e3-47ff-a6a7-b451b631146b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:50.457260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574291ec-82e3-47ff-a6a7-b451b631146b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:50.580529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 887/1778] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:50 2022] POST /identity/v3/projects => generated 399 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:50.584603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e9bc3c-88a7-41b4-8e1a-0e4625f86550 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:50.595873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e9bc3c-88a7-41b4-8e1a-0e4625f86550 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:50.656389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f38e45-211a-43f4-8d93-99f780b25115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:50.657012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f38e45-211a-43f4-8d93-99f780b25115 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:50.657313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f38e45-211a-43f4-8d93-99f780b25115 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:50.667867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86f38e45-211a-43f4-8d93-99f780b25115 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:50.667867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f38e45-211a-43f4-8d93-99f780b25115 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:50.667867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86f38e45-211a-43f4-8d93-99f780b25115 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:50.779237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 892/1779] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:50 2022] POST /identity/v3/users => generated 306 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:50.783408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574291ec-82e3-47ff-a6a7-b451b631146b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:50.797137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574291ec-82e3-47ff-a6a7-b451b631146b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:50.798471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9129dc2e-5ba9-4178-b0a4-4be474af5be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:50.799122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9129dc2e-5ba9-4178-b0a4-4be474af5be5 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:50.799459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9129dc2e-5ba9-4178-b0a4-4be474af5be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:50.799732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9129dc2e-5ba9-4178-b0a4-4be474af5be5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:50.800401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9129dc2e-5ba9-4178-b0a4-4be474af5be5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:50.801995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9129dc2e-5ba9-4178-b0a4-4be474af5be5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:50.816022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 888/1780] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:50.817578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f38e45-211a-43f4-8d93-99f780b25115 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:50.828611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86f38e45-211a-43f4-8d93-99f780b25115 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:50.833603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66597577-fbb0-4021-b189-56867c630dfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:50.834268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66597577-fbb0-4021-b189-56867c630dfb admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:50.834560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66597577-fbb0-4021-b189-56867c630dfb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:50.835137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66597577-fbb0-4021-b189-56867c630dfb admin admin] PATH_INFO: `/v3/projects/d40b62a2810242ddac2ed13c2835f754/users/5684ccbca2514a33938844acef8985e0/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:50.852571 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66597577-fbb0-4021-b189-56867c630dfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=d40b62a2810242ddac2ed13c2835f754, user_id=5684ccbca2514a33938844acef8985e0, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:50.855358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66597577-fbb0-4021-b189-56867c630dfb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:50.950230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:50.957828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 893/1781] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:50 2022] PUT /identity/v3/projects/d40b62a2810242ddac2ed13c2835f754/users/5684ccbca2514a33938844acef8985e0/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:50.961802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9129dc2e-5ba9-4178-b0a4-4be474af5be5 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:50.971719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9129dc2e-5ba9-4178-b0a4-4be474af5be5 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:50.980091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:51.028568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af9a9b9-52a4-4fa4-944b-c6d502614405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:51.029172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9a9b9-52a4-4fa4-944b-c6d502614405 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:51.029457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9a9b9-52a4-4fa4-944b-c6d502614405 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:51.029732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9a9b9-52a4-4fa4-944b-c6d502614405 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:51.030371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af9a9b9-52a4-4fa4-944b-c6d502614405 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:51.032013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af9a9b9-52a4-4fa4-944b-c6d502614405 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:51.041944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 889/1782] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:50 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:51.046217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66597577-fbb0-4021-b189-56867c630dfb admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:51.057475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66597577-fbb0-4021-b189-56867c630dfb admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:51.062855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:51.063681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:51.064173 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:51.064538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] PATH_INFO: `/v3/projects/d40b62a2810242ddac2ed13c2835f754/users/5684ccbca2514a33938844acef8985e0/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:51.067094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d40b62a2810242ddac2ed13c2835f754, user_id=5684ccbca2514a33938844acef8985e0, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:51.069787 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:51.081323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 894/1783] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:51 2022] PUT /identity/v3/projects/d40b62a2810242ddac2ed13c2835f754/users/5684ccbca2514a33938844acef8985e0/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:51.091752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11051da0-13ea-4233-807f-dc3477644ee4 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:51.092508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11051da0-13ea-4233-807f-dc3477644ee4 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:51.092508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11051da0-13ea-4233-807f-dc3477644ee4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:51.122134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-11051da0-13ea-4233-807f-dc3477644ee4 None None] MFA Rules not processed for user `5684ccbca2514a33938844acef8985e0`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:51.123420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-11051da0-13ea-4233-807f-dc3477644ee4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:51.200578 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 890/1784] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:51 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:51.211740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:51.213995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:51.276644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef291bab-c055-4a5f-bf4f-ab8ebcf9c74e admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:51.280620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:51.281183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:51.281520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:51.281796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:51.294272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:51.296375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:51.308913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 895/1785] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:51 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:03:51.658084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11051da0-13ea-4233-807f-dc3477644ee4 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:51.674353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11051da0-13ea-4233-807f-dc3477644ee4 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:51.677958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a076db73-e6ce-483c-9484-7c7330783f0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:51.678823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a076db73-e6ce-483c-9484-7c7330783f0b service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:51.679136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a076db73-e6ce-483c-9484-7c7330783f0b service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:51.679410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a076db73-e6ce-483c-9484-7c7330783f0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:51.736479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a076db73-e6ce-483c-9484-7c7330783f0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:51.737841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a076db73-e6ce-483c-9484-7c7330783f0b service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:51.746872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 891/1786] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:51 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:56.724246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:56.727431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:56.852436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dff0c2f-1de6-4a8d-b94c-f417710cd8a2 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:56.857586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:56.858346 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:56.858727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:56.859140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] PATH_INFO: `/v3/users/56b4866c6011456a8275cf84956373ca` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:56.860562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=56b4866c6011456a8275cf84956373ca)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:56.862712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:56.936711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a076db73-e6ce-483c-9484-7c7330783f0b service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:56.949076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a076db73-e6ce-483c-9484-7c7330783f0b service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:56.954082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:56.954661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:56.954941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:56.955277 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] PATH_INFO: `/v3/users/1a464fd6d93641f1b1c4f3d7bbf29b91` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:56.956444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a464fd6d93641f1b1c4f3d7bbf29b91)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:56.958372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:57.078170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:57.104396 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:57.114301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56b4866c6011456a8275cf84956373ca'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.155261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56b4866c6011456a8275cf84956373ca'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.157637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a464fd6d93641f1b1c4f3d7bbf29b91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.159234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a464fd6d93641f1b1c4f3d7bbf29b91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.168500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56b4866c6011456a8275cf84956373ca'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.172449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a464fd6d93641f1b1c4f3d7bbf29b91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.175849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56b4866c6011456a8275cf84956373ca'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.176632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-117b8407-6d17-4cd7-ba62-04b7b71e1e55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56b4866c6011456a8275cf84956373ca'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.180166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a464fd6d93641f1b1c4f3d7bbf29b91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.180836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a464fd6d93641f1b1c4f3d7bbf29b91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.183939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 896/1787] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:56 2022] DELETE /identity/v3/users/56b4866c6011456a8275cf84956373ca => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:57.262926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 892/1788] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:56 2022] DELETE /identity/v3/users/1a464fd6d93641f1b1c4f3d7bbf29b91 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:57.268311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9ef079-0677-4762-b51c-6d244d9bf113 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:57.269041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9ef079-0677-4762-b51c-6d244d9bf113 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:57.269041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9ef079-0677-4762-b51c-6d244d9bf113 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:57.291461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-9b9ef079-0677-4762-b51c-6d244d9bf113 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:57.293195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9b9ef079-0677-4762-b51c-6d244d9bf113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:57.367012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 897/1789] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:57.377228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:57.378795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:57.442544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9ef079-0677-4762-b51c-6d244d9bf113 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:57.444148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9b9ef079-0677-4762-b51c-6d244d9bf113 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:57.448983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29692de-3b1f-4fd6-95e8-3e75d2aa722d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:57.453896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b30a7b-11f2-4b71-a1c1-ca42002eb1c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:57.454521 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b30a7b-11f2-4b71-a1c1-ca42002eb1c6 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:57.454798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b30a7b-11f2-4b71-a1c1-ca42002eb1c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:57.455104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b30a7b-11f2-4b71-a1c1-ca42002eb1c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:57.467951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b30a7b-11f2-4b71-a1c1-ca42002eb1c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:57.468871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9ef079-0677-4762-b51c-6d244d9bf113 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:57.469908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b30a7b-11f2-4b71-a1c1-ca42002eb1c6 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:57.475975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:57.477851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:57.478497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:57.479446 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] PATH_INFO: `/v3/projects/ec86749bbe5743d08c6b3c2ae8eade76` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:57.481186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=ec86749bbe5743d08c6b3c2ae8eade76)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:57.483895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 893/1790] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:03:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:03:57.484931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:57.631848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ec86749bbe5743d08c6b3c2ae8eade76', 'request_id': 'req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.707794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ec86749bbe5743d08c6b3c2ae8eade76', 'request_id': 'req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.737912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ec86749bbe5743d08c6b3c2ae8eade76', 'request_id': 'req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.765006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] The token cache is being invalidate because project ec86749bbe5743d08c6b3c2ae8eade76 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:57.766404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.767096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:57.775401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 898/1791] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:57 2022] DELETE /identity/v3/projects/ec86749bbe5743d08c6b3c2ae8eade76 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:57.776690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b30a7b-11f2-4b71-a1c1-ca42002eb1c6 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:57.797057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b30a7b-11f2-4b71-a1c1-ca42002eb1c6 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:57.872488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:57.873050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:57.873327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:57.873806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] PATH_INFO: `/v3/users/5be43f8aa983469e8fa7ff4352004582` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:57.875104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5be43f8aa983469e8fa7ff4352004582)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:57.877153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:58.042117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5be43f8aa983469e8fa7ff4352004582'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.042867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5be43f8aa983469e8fa7ff4352004582'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.044272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:58.066462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5be43f8aa983469e8fa7ff4352004582'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.074395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5be43f8aa983469e8fa7ff4352004582'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.075217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5be43f8aa983469e8fa7ff4352004582'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.139849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:58.141321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:58.161261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75092af3-1838-4b2a-abfd-bc9a41cb3d4a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:58.170801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 894/1792] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:57 2022] DELETE /identity/v3/users/5be43f8aa983469e8fa7ff4352004582 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:58.175392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:58.176236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:58.176520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:58.176963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] PATH_INFO: `/v3/projects/9ed921dbf8b14e46a488ba4b7f412087` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:58.178416 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ed921dbf8b14e46a488ba4b7f412087)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:58.180482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:58.309121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:58.325001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ed921dbf8b14e46a488ba4b7f412087', 'request_id': 'req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.448286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ed921dbf8b14e46a488ba4b7f412087', 'request_id': 'req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.456360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:58.457819 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:58.458085 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ed921dbf8b14e46a488ba4b7f412087', 'request_id': 'req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.474155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] The token cache is being invalidate because project 9ed921dbf8b14e46a488ba4b7f412087 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:58.474447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.475211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.477003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 899/1793] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:58 2022] DELETE /identity/v3/projects/9ed921dbf8b14e46a488ba4b7f412087 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:58.480872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:58.484072 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:58.543492 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09566be7-d88c-49cd-ae90-e794dacc35d8 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:58.568648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0089d6f8-a91c-4dfd-9e2b-8d2a9f662131 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:58.575844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:58.576596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:58.577050 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:58.577462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] PATH_INFO: `/v3/users/5684ccbca2514a33938844acef8985e0` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:58.578681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5684ccbca2514a33938844acef8985e0)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:58.580867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:58.612181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:58.612968 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:58.613331 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:58.613610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] PATH_INFO: `/v3/projects/d4b4cb4efb26452496accb58164b8f21` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:58.614771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4b4cb4efb26452496accb58164b8f21)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:58.616708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:58.751230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5684ccbca2514a33938844acef8985e0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.751772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4b4cb4efb26452496accb58164b8f21', 'request_id': 'req-96c59777-b310-4610-8113-f0e69d8e61a7'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.757694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4b4cb4efb26452496accb58164b8f21', 'request_id': 'req-96c59777-b310-4610-8113-f0e69d8e61a7'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.763902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4b4cb4efb26452496accb58164b8f21', 'request_id': 'req-96c59777-b310-4610-8113-f0e69d8e61a7'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.828669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5684ccbca2514a33938844acef8985e0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.841889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5684ccbca2514a33938844acef8985e0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.849067 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5684ccbca2514a33938844acef8985e0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.849881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-2b0b94e7-0f34-4ded-aa95-eeaa34281063 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5684ccbca2514a33938844acef8985e0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.856386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 900/1794] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:03:58 2022] DELETE /identity/v3/users/5684ccbca2514a33938844acef8985e0 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:58.921510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] The token cache is being invalidate because project d4b4cb4efb26452496accb58164b8f21 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:58.921510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.922292 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-96c59777-b310-4610-8113-f0e69d8e61a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:58.924144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 895/1795] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:58 2022] DELETE /identity/v3/projects/d4b4cb4efb26452496accb58164b8f21 => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:58.930122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31903094-dc0b-491c-8f5b-e23c16c9d9ff None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:58.930436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31903094-dc0b-491c-8f5b-e23c16c9d9ff None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:58.931558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31903094-dc0b-491c-8f5b-e23c16c9d9ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:58.954393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-31903094-dc0b-491c-8f5b-e23c16c9d9ff None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:58.955752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-31903094-dc0b-491c-8f5b-e23c16c9d9ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:59.017087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 901/1796] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:59.025912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68110f-0a48-4808-89ac-7651ab81038d None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.026289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68110f-0a48-4808-89ac-7651ab81038d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.026564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68110f-0a48-4808-89ac-7651ab81038d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.034342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:59.041535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:59.047503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-1f68110f-0a48-4808-89ac-7651ab81038d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:59.048748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1f68110f-0a48-4808-89ac-7651ab81038d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:59.058767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 896/1797] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:03:59.067620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31903094-dc0b-491c-8f5b-e23c16c9d9ff None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.077283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31903094-dc0b-491c-8f5b-e23c16c9d9ff None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.082541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.083248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.083564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.083893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.084681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:59.084997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.086617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.102055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 902/1798] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:03:59.104192 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f68110f-0a48-4808-89ac-7651ab81038d None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.114022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f68110f-0a48-4808-89ac-7651ab81038d None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.126740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.126740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.126740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.126740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.127139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.127139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.220350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 897/1799] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/projects => generated 377 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:59.224489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.237755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0e34dba-b25b-4e34-b52e-8c8276cec3f6 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.257134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.258910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:59.313110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccdf75c-b795-4ccb-a31a-723e8d532b53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.313962 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccdf75c-b795-4ccb-a31a-723e8d532b53 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.314300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccdf75c-b795-4ccb-a31a-723e8d532b53 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.314582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccdf75c-b795-4ccb-a31a-723e8d532b53 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.315284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccdf75c-b795-4ccb-a31a-723e8d532b53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.317259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccdf75c-b795-4ccb-a31a-723e8d532b53 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.344833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e2b240-5239-4f24-84c3-c5073402dec1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.349790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.350417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.350694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.350988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] PATH_INFO: `/v3/projects/d40b62a2810242ddac2ed13c2835f754` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.352129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d40b62a2810242ddac2ed13c2835f754)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.354689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.384883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 903/1800] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:03:59.390244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccdf75c-b795-4ccb-a31a-723e8d532b53 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.402015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccdf75c-b795-4ccb-a31a-723e8d532b53 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.406810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c437a2-d072-4121-a8da-3d050107cb63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.407439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c437a2-d072-4121-a8da-3d050107cb63 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.417998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c437a2-d072-4121-a8da-3d050107cb63 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.417998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c437a2-d072-4121-a8da-3d050107cb63 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.417998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c437a2-d072-4121-a8da-3d050107cb63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.417998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01c437a2-d072-4121-a8da-3d050107cb63 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.421148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 904/1801] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:03:59.425032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c437a2-d072-4121-a8da-3d050107cb63 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.435411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c437a2-d072-4121-a8da-3d050107cb63 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.441189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fcaf9c-abb2-4017-81e9-85794bd56ae9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.441189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fcaf9c-abb2-4017-81e9-85794bd56ae9 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.441394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fcaf9c-abb2-4017-81e9-85794bd56ae9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.441538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8fcaf9c-abb2-4017-81e9-85794bd56ae9 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.442187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fcaf9c-abb2-4017-81e9-85794bd56ae9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.443836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8fcaf9c-abb2-4017-81e9-85794bd56ae9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.455437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 905/1802] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:03:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:03:59.459561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fcaf9c-abb2-4017-81e9-85794bd56ae9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.470211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8fcaf9c-abb2-4017-81e9-85794bd56ae9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.477279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd40b62a2810242ddac2ed13c2835f754', 'request_id': 'req-16a49c0e-d5bf-4e39-9c49-344abec91830'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:59.484455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd40b62a2810242ddac2ed13c2835f754', 'request_id': 'req-16a49c0e-d5bf-4e39-9c49-344abec91830'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:59.490275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd40b62a2810242ddac2ed13c2835f754', 'request_id': 'req-16a49c0e-d5bf-4e39-9c49-344abec91830'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:59.531535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2211efc3-4a9b-4226-8747-0e3980498d95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.532204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2211efc3-4a9b-4226-8747-0e3980498d95 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.532626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2211efc3-4a9b-4226-8747-0e3980498d95 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.532740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2211efc3-4a9b-4226-8747-0e3980498d95 admin admin] PATH_INFO: `/v3/projects/9fb0ecaa29134499a7f37e48ed59fc72/users/6f1e7de6e1364136930396fff0444752/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.551311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2211efc3-4a9b-4226-8747-0e3980498d95 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fb0ecaa29134499a7f37e48ed59fc72, user_id=6f1e7de6e1364136930396fff0444752, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.554339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2211efc3-4a9b-4226-8747-0e3980498d95 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.555854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] The token cache is being invalidate because project d40b62a2810242ddac2ed13c2835f754 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:03:59.556157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:59.556781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16a49c0e-d5bf-4e39-9c49-344abec91830 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:03:59.558333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 898/1803] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:03:59 2022] DELETE /identity/v3/projects/d40b62a2810242ddac2ed13c2835f754 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:03:59.563022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9235759-e18e-41c8-bf63-adca05694f6c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.563313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9235759-e18e-41c8-bf63-adca05694f6c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.563622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9235759-e18e-41c8-bf63-adca05694f6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.583145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c9235759-e18e-41c8-bf63-adca05694f6c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:59.584384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c9235759-e18e-41c8-bf63-adca05694f6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:59.593233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 899/1804] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:59.600827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 906/1805] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:03:59 2022] PUT /identity/v3/projects/9fb0ecaa29134499a7f37e48ed59fc72/users/6f1e7de6e1364136930396fff0444752/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:03:59.601691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f4424-b0aa-409e-b640-69d93b1ec68a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.601978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f4424-b0aa-409e-b640-69d93b1ec68a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.606120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e69f4424-b0aa-409e-b640-69d93b1ec68a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.615388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223742c6-38ae-45e0-9ea6-ac4f689714f3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.615668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223742c6-38ae-45e0-9ea6-ac4f689714f3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.615942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223742c6-38ae-45e0-9ea6-ac4f689714f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.623039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e69f4424-b0aa-409e-b640-69d93b1ec68a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:59.624245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e69f4424-b0aa-409e-b640-69d93b1ec68a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:59.644791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-223742c6-38ae-45e0-9ea6-ac4f689714f3 None None] MFA Rules not processed for user `6f1e7de6e1364136930396fff0444752`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:03:59.646002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-223742c6-38ae-45e0-9ea6-ac4f689714f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:59.650556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:59.676019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:03:59.686944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 900/1806] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:59.691221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69f4424-b0aa-409e-b640-69d93b1ec68a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.704535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e69f4424-b0aa-409e-b640-69d93b1ec68a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.713092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.713753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.713985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.714253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.714910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:03:59.715375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.717043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.731005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 901/1807] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:03:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:03:59.731650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 907/1808] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:03:59.735886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.737310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223742c6-38ae-45e0-9ea6-ac4f689714f3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.737841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:03:59.748368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223742c6-38ae-45e0-9ea6-ac4f689714f3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.750298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cf71e0a-1f26-4d24-893e-2ebdb0bfbfdc admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.752910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be112e-41e8-4b10-92f4-e9d4dd8f6852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.753638 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be112e-41e8-4b10-92f4-e9d4dd8f6852 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.753919 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be112e-41e8-4b10-92f4-e9d4dd8f6852 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.754191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26be112e-41e8-4b10-92f4-e9d4dd8f6852 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.754858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26be112e-41e8-4b10-92f4-e9d4dd8f6852 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.757518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41e0dc3-0625-43ff-8833-3e4b017b6239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.758113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e0dc3-0625-43ff-8833-3e4b017b6239 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.758214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26be112e-41e8-4b10-92f4-e9d4dd8f6852 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.758396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e0dc3-0625-43ff-8833-3e4b017b6239 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.758672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41e0dc3-0625-43ff-8833-3e4b017b6239 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.759852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41e0dc3-0625-43ff-8833-3e4b017b6239 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.761746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41e0dc3-0625-43ff-8833-3e4b017b6239 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:03:59.853491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 908/1809] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/projects => generated 365 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:59.857838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be112e-41e8-4b10-92f4-e9d4dd8f6852 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.902571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 902/1810] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/projects => generated 377 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:03:59.904414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41e0dc3-0625-43ff-8833-3e4b017b6239 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:03:59.937232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26be112e-41e8-4b10-92f4-e9d4dd8f6852 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.983182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41e0dc3-0625-43ff-8833-3e4b017b6239 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:03:59.988263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc60e4-c0a2-453a-baf2-39206634a7ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:03:59.988880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc60e4-c0a2-453a-baf2-39206634a7ec admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:03:59.989170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc60e4-c0a2-453a-baf2-39206634a7ec admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:03:59.989441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3fc60e4-c0a2-453a-baf2-39206634a7ec admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:03:59.990079 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fc60e4-c0a2-453a-baf2-39206634a7ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:03:59.992061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3fc60e4-c0a2-453a-baf2-39206634a7ec admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.003198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73700cbd-464a-4e58-954b-4873c2d4c610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.003908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73700cbd-464a-4e58-954b-4873c2d4c610 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.004290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73700cbd-464a-4e58-954b-4873c2d4c610 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.005158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73700cbd-464a-4e58-954b-4873c2d4c610 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.007828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73700cbd-464a-4e58-954b-4873c2d4c610 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.014722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73700cbd-464a-4e58-954b-4873c2d4c610 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.090288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 903/1811] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/users => generated 289 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:00.091552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc60e4-c0a2-453a-baf2-39206634a7ec admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.101660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3fc60e4-c0a2-453a-baf2-39206634a7ec admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.111228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 909/1812] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:03:59 2022] POST /identity/v3/users => generated 283 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:04:00.115553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73700cbd-464a-4e58-954b-4873c2d4c610 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.118590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1b5aac-9624-4128-b884-ecb983fe0f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.119260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1b5aac-9624-4128-b884-ecb983fe0f33 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.119520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1b5aac-9624-4128-b884-ecb983fe0f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.119799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1b5aac-9624-4128-b884-ecb983fe0f33 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.120459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1b5aac-9624-4128-b884-ecb983fe0f33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.122075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1b5aac-9624-4128-b884-ecb983fe0f33 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.126296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73700cbd-464a-4e58-954b-4873c2d4c610 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.131319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889f05c3-a50d-4f12-881b-9d7e02b7da2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.132049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889f05c3-a50d-4f12-881b-9d7e02b7da2b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.132437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889f05c3-a50d-4f12-881b-9d7e02b7da2b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.132868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889f05c3-a50d-4f12-881b-9d7e02b7da2b admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.133783 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 904/1813] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:00.137226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889f05c3-a50d-4f12-881b-9d7e02b7da2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.138174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1b5aac-9624-4128-b884-ecb983fe0f33 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.139164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889f05c3-a50d-4f12-881b-9d7e02b7da2b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.148806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1b5aac-9624-4128-b884-ecb983fe0f33 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.153874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 910/1814] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:00.154710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f42846-3253-48b1-9a59-49a272c98324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.155702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f42846-3253-48b1-9a59-49a272c98324 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.156058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f42846-3253-48b1-9a59-49a272c98324 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.158062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f42846-3253-48b1-9a59-49a272c98324 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.158062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f42846-3253-48b1-9a59-49a272c98324 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.158242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889f05c3-a50d-4f12-881b-9d7e02b7da2b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.159114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f42846-3253-48b1-9a59-49a272c98324 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.168807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889f05c3-a50d-4f12-881b-9d7e02b7da2b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.170849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 905/1815] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:00.175408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f42846-3253-48b1-9a59-49a272c98324 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.177934 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef55a394-39d2-4b8c-9829-ce7052235d2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.178819 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef55a394-39d2-4b8c-9829-ce7052235d2b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.179178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef55a394-39d2-4b8c-9829-ce7052235d2b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.181445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef55a394-39d2-4b8c-9829-ce7052235d2b admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.181445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef55a394-39d2-4b8c-9829-ce7052235d2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.183140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef55a394-39d2-4b8c-9829-ce7052235d2b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.186449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:00.187572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f42846-3253-48b1-9a59-49a272c98324 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.192778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83c7317-dd47-4243-8a63-50ae968575cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.193515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c7317-dd47-4243-8a63-50ae968575cd admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.193857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c7317-dd47-4243-8a63-50ae968575cd admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.194195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83c7317-dd47-4243-8a63-50ae968575cd admin admin] PATH_INFO: `/v3/projects/65c5ed4451c64fa2b15e98e0e13d60b1/users/646561c045bd4b21b9d95bd0e8ebcc45/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.197241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 911/1816] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:00.202937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef55a394-39d2-4b8c-9829-ce7052235d2b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.214462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83c7317-dd47-4243-8a63-50ae968575cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=65c5ed4451c64fa2b15e98e0e13d60b1, user_id=646561c045bd4b21b9d95bd0e8ebcc45, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.215610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef55a394-39d2-4b8c-9829-ce7052235d2b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.217603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83c7317-dd47-4243-8a63-50ae968575cd admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.221743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349d56eb-376f-4b1a-b6d6-b9faa09c27a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.222621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d56eb-376f-4b1a-b6d6-b9faa09c27a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.222746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d56eb-376f-4b1a-b6d6-b9faa09c27a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.223084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d56eb-376f-4b1a-b6d6-b9faa09c27a6 admin admin] PATH_INFO: `/v3/projects/c64b8518083b460cac31423afdeb768a/users/ae4d0cec46934f95bf8187744db15e9e/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.238313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349d56eb-376f-4b1a-b6d6-b9faa09c27a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c64b8518083b460cac31423afdeb768a, user_id=ae4d0cec46934f95bf8187744db15e9e, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.241152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349d56eb-376f-4b1a-b6d6-b9faa09c27a6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.303762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 906/1817] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:00 2022] PUT /identity/v3/projects/65c5ed4451c64fa2b15e98e0e13d60b1/users/646561c045bd4b21b9d95bd0e8ebcc45/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:00.315594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e911b-36ff-4bf5-80f8-de28aca40366 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.315876 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e911b-36ff-4bf5-80f8-de28aca40366 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.316149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649e911b-36ff-4bf5-80f8-de28aca40366 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.334471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 912/1818] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:00 2022] PUT /identity/v3/projects/c64b8518083b460cac31423afdeb768a/users/ae4d0cec46934f95bf8187744db15e9e/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:00.344716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a3eb2-3ed0-4adf-b506-0ef73149df7a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.345009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a3eb2-3ed0-4adf-b506-0ef73149df7a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.345267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5a3eb2-3ed0-4adf-b506-0ef73149df7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.346579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-649e911b-36ff-4bf5-80f8-de28aca40366 None None] MFA Rules not processed for user `646561c045bd4b21b9d95bd0e8ebcc45`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:00.347912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-649e911b-36ff-4bf5-80f8-de28aca40366 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:00.373727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-2f5a3eb2-3ed0-4adf-b506-0ef73149df7a None None] MFA Rules not processed for user `ae4d0cec46934f95bf8187744db15e9e`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:00.374867 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2f5a3eb2-3ed0-4adf-b506-0ef73149df7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:00.414400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:00.433691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 907/1819] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:00 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:04:00.439565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649e911b-36ff-4bf5-80f8-de28aca40366 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.490686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 913/1820] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:00 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:04:00.490686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a3eb2-3ed0-4adf-b506-0ef73149df7a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.533477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649e911b-36ff-4bf5-80f8-de28aca40366 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.539001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ee88eb-36e0-4134-b72e-35a757409a3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.539366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ee88eb-36e0-4134-b72e-35a757409a3c admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.539645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ee88eb-36e0-4134-b72e-35a757409a3c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.539919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ee88eb-36e0-4134-b72e-35a757409a3c admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.542193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ee88eb-36e0-4134-b72e-35a757409a3c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.544463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ee88eb-36e0-4134-b72e-35a757409a3c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.577872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5a3eb2-3ed0-4adf-b506-0ef73149df7a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.583983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4573a6-504f-4515-aef8-e4692d33d1a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.584604 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4573a6-504f-4515-aef8-e4692d33d1a9 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.584887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4573a6-504f-4515-aef8-e4692d33d1a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.585162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4573a6-504f-4515-aef8-e4692d33d1a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.585928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4573a6-504f-4515-aef8-e4692d33d1a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.591122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4573a6-504f-4515-aef8-e4692d33d1a9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.650992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 908/1821] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:04:00 2022] POST /identity/v3/projects => generated 375 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:00.652884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ee88eb-36e0-4134-b72e-35a757409a3c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.707817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 914/1822] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:04:00 2022] POST /identity/v3/projects => generated 367 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:00.711795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4573a6-504f-4515-aef8-e4692d33d1a9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.728668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ee88eb-36e0-4134-b72e-35a757409a3c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.782832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:00.802764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e3578-b0b8-436c-b4f2-a8e5702504c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.803673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e3578-b0b8-436c-b4f2-a8e5702504c1 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.803949 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e3578-b0b8-436c-b4f2-a8e5702504c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.804217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d80e3578-b0b8-436c-b4f2-a8e5702504c1 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.804593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4573a6-504f-4515-aef8-e4692d33d1a9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.804921 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80e3578-b0b8-436c-b4f2-a8e5702504c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.807426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d80e3578-b0b8-436c-b4f2-a8e5702504c1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.810216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b23c8b2-f283-458e-8d90-1cf187cb0c93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.810880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b23c8b2-f283-458e-8d90-1cf187cb0c93 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.811489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b23c8b2-f283-458e-8d90-1cf187cb0c93 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.812018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b23c8b2-f283-458e-8d90-1cf187cb0c93 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.813105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b23c8b2-f283-458e-8d90-1cf187cb0c93 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.815087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b23c8b2-f283-458e-8d90-1cf187cb0c93 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.909306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 909/1823] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:04:00 2022] POST /identity/v3/users => generated 294 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:04:00.913064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e3578-b0b8-436c-b4f2-a8e5702504c1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.923253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d80e3578-b0b8-436c-b4f2-a8e5702504c1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.927732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a735f37-8328-4f5a-a4c6-83ae4ecd0968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.928548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a735f37-8328-4f5a-a4c6-83ae4ecd0968 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.928840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a735f37-8328-4f5a-a4c6-83ae4ecd0968 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.929148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a735f37-8328-4f5a-a4c6-83ae4ecd0968 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.929776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a735f37-8328-4f5a-a4c6-83ae4ecd0968 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.931455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a735f37-8328-4f5a-a4c6-83ae4ecd0968 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.940807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 915/1824] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:04:00 2022] POST /identity/v3/users => generated 290 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:00.944550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 910/1825] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:00.944673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b23c8b2-f283-458e-8d90-1cf187cb0c93 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.945758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a735f37-8328-4f5a-a4c6-83ae4ecd0968 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.951926 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b23c8b2-f283-458e-8d90-1cf187cb0c93 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.955623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a735f37-8328-4f5a-a4c6-83ae4ecd0968 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.956499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cfe9d5-c741-4a50-b85d-3879a3426f5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.959104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfe9d5-c741-4a50-b85d-3879a3426f5e admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.959104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfe9d5-c741-4a50-b85d-3879a3426f5e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.959104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cfe9d5-c741-4a50-b85d-3879a3426f5e admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.959104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cfe9d5-c741-4a50-b85d-3879a3426f5e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.959872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3cfe9d5-c741-4a50-b85d-3879a3426f5e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.960371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465a8fbb-6a8b-4701-9a74-2e220e1c2b30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.961031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a8fbb-6a8b-4701-9a74-2e220e1c2b30 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.961314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a8fbb-6a8b-4701-9a74-2e220e1c2b30 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.961592 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465a8fbb-6a8b-4701-9a74-2e220e1c2b30 admin admin] PATH_INFO: `/v3/projects/d4067053a5a340338bfbd7b36f90e311/users/086a08be2c3045c49e38de8f315df67a/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:00.964531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:00.970985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 916/1826] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:00 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:00.975074 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cfe9d5-c741-4a50-b85d-3879a3426f5e admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:00.978897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465a8fbb-6a8b-4701-9a74-2e220e1c2b30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4067053a5a340338bfbd7b36f90e311, user_id=086a08be2c3045c49e38de8f315df67a, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:00.981626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465a8fbb-6a8b-4701-9a74-2e220e1c2b30 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:00.985508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3cfe9d5-c741-4a50-b85d-3879a3426f5e admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:00.990108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dd318b-3ee2-47fc-8f7f-11b89a084ee5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:00.990733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dd318b-3ee2-47fc-8f7f-11b89a084ee5 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:00.991042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dd318b-3ee2-47fc-8f7f-11b89a084ee5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:00.991314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dd318b-3ee2-47fc-8f7f-11b89a084ee5 admin admin] PATH_INFO: `/v3/projects/938f6c43270c4a85b415b22bc9445c25/users/5edc51ae1f1c4ca1af0d5f4b76f6b5c9/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.005820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dd318b-3ee2-47fc-8f7f-11b89a084ee5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=938f6c43270c4a85b415b22bc9445c25, user_id=5edc51ae1f1c4ca1af0d5f4b76f6b5c9, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:01.008471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68dd318b-3ee2-47fc-8f7f-11b89a084ee5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:01.064155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 911/1827] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:00 2022] PUT /identity/v3/projects/d4067053a5a340338bfbd7b36f90e311/users/086a08be2c3045c49e38de8f315df67a/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:04:01.068436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465a8fbb-6a8b-4701-9a74-2e220e1c2b30 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:01.081108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 917/1828] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:00 2022] PUT /identity/v3/projects/938f6c43270c4a85b415b22bc9445c25/users/5edc51ae1f1c4ca1af0d5f4b76f6b5c9/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:04:01.085701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dd318b-3ee2-47fc-8f7f-11b89a084ee5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:01.134736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465a8fbb-6a8b-4701-9a74-2e220e1c2b30 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:01.151640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68dd318b-3ee2-47fc-8f7f-11b89a084ee5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:01.156713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7eb7f3-7e0b-41c5-9b10-828385fc5c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:01.157320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7eb7f3-7e0b-41c5-9b10-828385fc5c28 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:01.157603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7eb7f3-7e0b-41c5-9b10-828385fc5c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:01.157878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7eb7f3-7e0b-41c5-9b10-828385fc5c28 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.158557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7eb7f3-7e0b-41c5-9b10-828385fc5c28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:01.160294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7eb7f3-7e0b-41c5-9b10-828385fc5c28 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:01.171158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 918/1829] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:01 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:01.175276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7eb7f3-7e0b-41c5-9b10-828385fc5c28 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:01.185371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7eb7f3-7e0b-41c5-9b10-828385fc5c28 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:01.190160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a8714f-20f7-4680-b0a3-d5858aa93d9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:01.190802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a8714f-20f7-4680-b0a3-d5858aa93d9d admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:01.191118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a8714f-20f7-4680-b0a3-d5858aa93d9d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:01.191398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a8714f-20f7-4680-b0a3-d5858aa93d9d admin admin] PATH_INFO: `/v3/projects/938f6c43270c4a85b415b22bc9445c25/users/5edc51ae1f1c4ca1af0d5f4b76f6b5c9/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.195094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a8714f-20f7-4680-b0a3-d5858aa93d9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=938f6c43270c4a85b415b22bc9445c25, user_id=5edc51ae1f1c4ca1af0d5f4b76f6b5c9, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:01.196632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea6dd19-0a71-47d2-8f5a-3d13412959d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:01.197295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea6dd19-0a71-47d2-8f5a-3d13412959d4 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:01.197539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea6dd19-0a71-47d2-8f5a-3d13412959d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:01.197818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea6dd19-0a71-47d2-8f5a-3d13412959d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.197941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a8714f-20f7-4680-b0a3-d5858aa93d9d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:01.198475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea6dd19-0a71-47d2-8f5a-3d13412959d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:01.200065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea6dd19-0a71-47d2-8f5a-3d13412959d4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:01.210886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 912/1830] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:01 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:01.211787 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 919/1831] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:01 2022] PUT /identity/v3/projects/938f6c43270c4a85b415b22bc9445c25/users/5edc51ae1f1c4ca1af0d5f4b76f6b5c9/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:04:01.216121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea6dd19-0a71-47d2-8f5a-3d13412959d4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:01.223403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dff6ef-705e-4df3-aff7-8799648b3d02 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:01.223693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dff6ef-705e-4df3-aff7-8799648b3d02 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:01.223967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dff6ef-705e-4df3-aff7-8799648b3d02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.226208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea6dd19-0a71-47d2-8f5a-3d13412959d4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:01.253030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-57dff6ef-705e-4df3-aff7-8799648b3d02 None None] MFA Rules not processed for user `5edc51ae1f1c4ca1af0d5f4b76f6b5c9`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:01.254255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-57dff6ef-705e-4df3-aff7-8799648b3d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:01.298818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3777de95-0005-4576-bfa1-536c4a5e0623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:01.299515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3777de95-0005-4576-bfa1-536c4a5e0623 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:01.299793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3777de95-0005-4576-bfa1-536c4a5e0623 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:01.300064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3777de95-0005-4576-bfa1-536c4a5e0623 admin admin] PATH_INFO: `/v3/projects/d4067053a5a340338bfbd7b36f90e311/users/086a08be2c3045c49e38de8f315df67a/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.302036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3777de95-0005-4576-bfa1-536c4a5e0623 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4067053a5a340338bfbd7b36f90e311, user_id=086a08be2c3045c49e38de8f315df67a, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:01.305097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3777de95-0005-4576-bfa1-536c4a5e0623 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:01.316544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 913/1832] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:01 2022] PUT /identity/v3/projects/d4067053a5a340338bfbd7b36f90e311/users/086a08be2c3045c49e38de8f315df67a/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:04:01.328601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c263c-c0c9-40ee-ace8-d32a273515e5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:01.328890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c263c-c0c9-40ee-ace8-d32a273515e5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:01.329166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c263c-c0c9-40ee-ace8-d32a273515e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.336548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 920/1833] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:01 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:04:01.353985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dff6ef-705e-4df3-aff7-8799648b3d02 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:01.355390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-57dff6ef-705e-4df3-aff7-8799648b3d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:01.361221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d96c263c-c0c9-40ee-ace8-d32a273515e5 None None] MFA Rules not processed for user `086a08be2c3045c49e38de8f315df67a`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:01.362428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d96c263c-c0c9-40ee-ace8-d32a273515e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:01.368966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:01.371357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:01.419693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57dff6ef-705e-4df3-aff7-8799648b3d02 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:01.434682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 914/1834] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:01 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:04:01.446225 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96c263c-c0c9-40ee-ace8-d32a273515e5 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:01.448536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d96c263c-c0c9-40ee-ace8-d32a273515e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:01.473225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac709a-6089-4115-bc9d-967463752465 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:01.473832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac709a-6089-4115-bc9d-967463752465 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:01.474164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac709a-6089-4115-bc9d-967463752465 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:01.474442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ac709a-6089-4115-bc9d-967463752465 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.506337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96c263c-c0c9-40ee-ace8-d32a273515e5 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:01.510853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fde7a0-eeaf-42d5-9503-449104e8d663 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:01.511513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fde7a0-eeaf-42d5-9503-449104e8d663 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:01.511844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fde7a0-eeaf-42d5-9503-449104e8d663 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:01.512123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fde7a0-eeaf-42d5-9503-449104e8d663 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:01.536963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ac709a-6089-4115-bc9d-967463752465 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:01.538452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ac709a-6089-4115-bc9d-967463752465 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:01.551295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 921/1835] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:01 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:04:01.573742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fde7a0-eeaf-42d5-9503-449104e8d663 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:01.575477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fde7a0-eeaf-42d5-9503-449104e8d663 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:01.585445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 915/1836] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:01 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:04:02.175081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac709a-6089-4115-bc9d-967463752465 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:02.185326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ac709a-6089-4115-bc9d-967463752465 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:02.189088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf03e91-2f8d-4527-a7dc-283cef04e2dd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:02.189630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf03e91-2f8d-4527-a7dc-283cef04e2dd service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:02.189905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf03e91-2f8d-4527-a7dc-283cef04e2dd service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:02.190202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf03e91-2f8d-4527-a7dc-283cef04e2dd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:02.250912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf03e91-2f8d-4527-a7dc-283cef04e2dd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:02.252389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf03e91-2f8d-4527-a7dc-283cef04e2dd service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:02.261739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 922/1837] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:02 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:03.873975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fde7a0-eeaf-42d5-9503-449104e8d663 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:03.884250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fde7a0-eeaf-42d5-9503-449104e8d663 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:03.887274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a57933-18c7-4e40-9e4e-02dbf60ea4d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:03.887972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a57933-18c7-4e40-9e4e-02dbf60ea4d9 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:03.888254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a57933-18c7-4e40-9e4e-02dbf60ea4d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:03.888554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a57933-18c7-4e40-9e4e-02dbf60ea4d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:03.946696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a57933-18c7-4e40-9e4e-02dbf60ea4d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:03.948215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a57933-18c7-4e40-9e4e-02dbf60ea4d9 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:03.951706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:03.957858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 916/1838] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:03 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:05.397236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf03e91-2f8d-4527-a7dc-283cef04e2dd service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:05.406671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf03e91-2f8d-4527-a7dc-283cef04e2dd service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:05.410392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca24694-2244-4470-a784-357ecbbd2087 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:05.411089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca24694-2244-4470-a784-357ecbbd2087 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:05.411373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca24694-2244-4470-a784-357ecbbd2087 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:05.411667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca24694-2244-4470-a784-357ecbbd2087 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:05.421159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca24694-2244-4470-a784-357ecbbd2087 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:05.422788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca24694-2244-4470-a784-357ecbbd2087 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:05.433865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 923/1839] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:05 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:09.232146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a57933-18c7-4e40-9e4e-02dbf60ea4d9 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:09.298475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a57933-18c7-4e40-9e4e-02dbf60ea4d9 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:09.303349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:09.303983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:09.304262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:09.304554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] PATH_INFO: `/v3/users/ae4d0cec46934f95bf8187744db15e9e` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:09.306201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae4d0cec46934f95bf8187744db15e9e)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:09.308205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:09.512139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae4d0cec46934f95bf8187744db15e9e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:09.512728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae4d0cec46934f95bf8187744db15e9e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:09.524748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae4d0cec46934f95bf8187744db15e9e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:09.531590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae4d0cec46934f95bf8187744db15e9e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:09.532255 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae4d0cec46934f95bf8187744db15e9e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:09.646011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 917/1840] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:04:09 2022] DELETE /identity/v3/users/ae4d0cec46934f95bf8187744db15e9e => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:09.650441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8577cea-75d0-4644-a4af-b0a446123c2c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:09.650888 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8577cea-75d0-4644-a4af-b0a446123c2c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:09.651039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8577cea-75d0-4644-a4af-b0a446123c2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:09.658210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:09.670972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b8577cea-75d0-4644-a4af-b0a446123c2c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:09.672186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b8577cea-75d0-4644-a4af-b0a446123c2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:09.730725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 924/1841] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:09.740604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:09.742052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:09.811660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf65332b-7ec6-4d8d-80eb-bf22dbfa9a90 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:09.815275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada93329-0f0e-4eb9-93c7-0212665556e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:09.815924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada93329-0f0e-4eb9-93c7-0212665556e0 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:09.816203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada93329-0f0e-4eb9-93c7-0212665556e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:09.816477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada93329-0f0e-4eb9-93c7-0212665556e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:09.825393 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada93329-0f0e-4eb9-93c7-0212665556e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:09.826814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada93329-0f0e-4eb9-93c7-0212665556e0 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:09.836357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 918/1842] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:10.274943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8577cea-75d0-4644-a4af-b0a446123c2c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:10.277353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b8577cea-75d0-4644-a4af-b0a446123c2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:10.293131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8577cea-75d0-4644-a4af-b0a446123c2c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:10.299943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:10.300787 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:10.301056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:10.301552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] PATH_INFO: `/v3/projects/c64b8518083b460cac31423afdeb768a` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:10.303633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c64b8518083b460cac31423afdeb768a)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:10.306093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:10.447199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c64b8518083b460cac31423afdeb768a', 'request_id': 'req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.526378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c64b8518083b460cac31423afdeb768a', 'request_id': 'req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.532315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c64b8518083b460cac31423afdeb768a', 'request_id': 'req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.543645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] The token cache is being invalidate because project c64b8518083b460cac31423afdeb768a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:04:10.543937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.544561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.546042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 925/1843] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:04:10 2022] DELETE /identity/v3/projects/c64b8518083b460cac31423afdeb768a => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:10.555941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada93329-0f0e-4eb9-93c7-0212665556e0 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:10.565539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada93329-0f0e-4eb9-93c7-0212665556e0 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:10.582181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:10.624089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:10.624722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:10.624999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:10.625270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] PATH_INFO: `/v3/users/5edc51ae1f1c4ca1af0d5f4b76f6b5c9` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:10.626380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5edc51ae1f1c4ca1af0d5f4b76f6b5c9)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:10.628471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:10.826000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5edc51ae1f1c4ca1af0d5f4b76f6b5c9'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.826590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5edc51ae1f1c4ca1af0d5f4b76f6b5c9'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.839743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5edc51ae1f1c4ca1af0d5f4b76f6b5c9'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.847068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5edc51ae1f1c4ca1af0d5f4b76f6b5c9'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.847782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-30e02c82-13cb-4e9a-8ce6-306f08747e99 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5edc51ae1f1c4ca1af0d5f4b76f6b5c9'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:10.925200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 919/1844] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:04:10 2022] DELETE /identity/v3/users/5edc51ae1f1c4ca1af0d5f4b76f6b5c9 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:11.361676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:11.363144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:11.377634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf8b6b78-2b8b-4e95-a4e1-f6734e750887 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:11.449723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:11.450366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:11.450644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:11.450924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] PATH_INFO: `/v3/projects/938f6c43270c4a85b415b22bc9445c25` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:11.452087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=938f6c43270c4a85b415b22bc9445c25)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:11.454000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:11.541122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:11.558097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '938f6c43270c4a85b415b22bc9445c25', 'request_id': 'req-4e72e61d-3160-42dc-877e-8be183cb3fe9'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:11.629083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '938f6c43270c4a85b415b22bc9445c25', 'request_id': 'req-4e72e61d-3160-42dc-877e-8be183cb3fe9'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:11.635138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '938f6c43270c4a85b415b22bc9445c25', 'request_id': 'req-4e72e61d-3160-42dc-877e-8be183cb3fe9'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:11.646710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] The token cache is being invalidate because project 938f6c43270c4a85b415b22bc9445c25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:04:11.647021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:11.647660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-4e72e61d-3160-42dc-877e-8be183cb3fe9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:11.649136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 926/1845] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:04:11 2022] DELETE /identity/v3/projects/938f6c43270c4a85b415b22bc9445c25 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:11.654594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce70a312-5390-4b70-a526-bf5e5eae960f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:11.655082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce70a312-5390-4b70-a526-bf5e5eae960f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:11.655267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce70a312-5390-4b70-a526-bf5e5eae960f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:11.675090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ce70a312-5390-4b70-a526-bf5e5eae960f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:11.676288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ce70a312-5390-4b70-a526-bf5e5eae960f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:11.737415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 920/1846] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:04:11.746846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d972753d-bade-46f2-8aeb-8f256ae38583 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:11.747191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d972753d-bade-46f2-8aeb-8f256ae38583 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:11.747473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d972753d-bade-46f2-8aeb-8f256ae38583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:11.772413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-d972753d-bade-46f2-8aeb-8f256ae38583 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:11.773626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d972753d-bade-46f2-8aeb-8f256ae38583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:11.783051 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 927/1847] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:11.787328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce70a312-5390-4b70-a526-bf5e5eae960f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:11.799362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce70a312-5390-4b70-a526-bf5e5eae960f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:11.804374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:11.804969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:11.805248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:11.805520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:11.806161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:04:11.806900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:11.808526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:11.820461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 921/1848] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:04:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:04:11.825211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d972753d-bade-46f2-8aeb-8f256ae38583 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:11.836084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d972753d-bade-46f2-8aeb-8f256ae38583 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:11.841892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48a72b2-e42e-47ff-a625-e9116e1723fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:11.842549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48a72b2-e42e-47ff-a625-e9116e1723fd admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:11.842862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48a72b2-e42e-47ff-a625-e9116e1723fd admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:11.843210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48a72b2-e42e-47ff-a625-e9116e1723fd admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:11.843893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48a72b2-e42e-47ff-a625-e9116e1723fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:11.845843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48a72b2-e42e-47ff-a625-e9116e1723fd admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:11.928473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 928/1849] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:04:11 2022] POST /identity/v3/projects => generated 375 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:11.932995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:11.943865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61154d02-fe13-4a87-9081-85104e873370 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:11.970676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:12.006080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d46fb60-7822-4498-8cf1-167c8cea0376 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:12.006779 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d46fb60-7822-4498-8cf1-167c8cea0376 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.007231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d46fb60-7822-4498-8cf1-167c8cea0376 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.014181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d46fb60-7822-4498-8cf1-167c8cea0376 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.014181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d46fb60-7822-4498-8cf1-167c8cea0376 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:12.014181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d46fb60-7822-4498-8cf1-167c8cea0376 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:12.093226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 922/1850] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:04:11 2022] POST /identity/v3/users => generated 288 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:12.097689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48a72b2-e42e-47ff-a625-e9116e1723fd admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:12.108953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48a72b2-e42e-47ff-a625-e9116e1723fd admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:12.114869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee4fa50-0524-4083-b419-f65ed3286989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:12.115537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee4fa50-0524-4083-b419-f65ed3286989 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.115817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee4fa50-0524-4083-b419-f65ed3286989 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.116214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee4fa50-0524-4083-b419-f65ed3286989 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.116877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee4fa50-0524-4083-b419-f65ed3286989 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:12.119130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee4fa50-0524-4083-b419-f65ed3286989 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:12.129492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 929/1851] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:12.139072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d46fb60-7822-4498-8cf1-167c8cea0376 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:12.149124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d46fb60-7822-4498-8cf1-167c8cea0376 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:12.154836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2d6bff-32cd-4b6f-baf5-58ec4f46b69d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:12.155463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2d6bff-32cd-4b6f-baf5-58ec4f46b69d admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.155743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2d6bff-32cd-4b6f-baf5-58ec4f46b69d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.156014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2d6bff-32cd-4b6f-baf5-58ec4f46b69d admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.156652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2d6bff-32cd-4b6f-baf5-58ec4f46b69d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:12.158229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2d6bff-32cd-4b6f-baf5-58ec4f46b69d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:12.169540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 923/1852] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:12.173668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee4fa50-0524-4083-b419-f65ed3286989 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:12.184885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee4fa50-0524-4083-b419-f65ed3286989 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:12.189858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa75336d-6e12-431c-a738-218f687612f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:12.190515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa75336d-6e12-431c-a738-218f687612f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.190789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa75336d-6e12-431c-a738-218f687612f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.191088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa75336d-6e12-431c-a738-218f687612f9 admin admin] PATH_INFO: `/v3/projects/a52addd3ae1b4f4dbbd05575db029d62/users/b7fb5f7fce6e48959f13cc17548a9964/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.207086 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa75336d-6e12-431c-a738-218f687612f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a52addd3ae1b4f4dbbd05575db029d62, user_id=b7fb5f7fce6e48959f13cc17548a9964, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:12.209909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa75336d-6e12-431c-a738-218f687612f9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:12.363112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 930/1853] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:12 2022] PUT /identity/v3/projects/a52addd3ae1b4f4dbbd05575db029d62/users/b7fb5f7fce6e48959f13cc17548a9964/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:12.375976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca0ff9a-8a3d-4843-973d-82c21b0d462a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.376663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca0ff9a-8a3d-4843-973d-82c21b0d462a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.377341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca0ff9a-8a3d-4843-973d-82c21b0d462a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.409670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-eca0ff9a-8a3d-4843-973d-82c21b0d462a None None] MFA Rules not processed for user `b7fb5f7fce6e48959f13cc17548a9964`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:12.410905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-eca0ff9a-8a3d-4843-973d-82c21b0d462a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:12.492499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 924/1854] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:12 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:12.497084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa75336d-6e12-431c-a738-218f687612f9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:12.562175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa75336d-6e12-431c-a738-218f687612f9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:12.567003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d10e955-a9a7-4fb3-ac66-4216342c40d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:12.567611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d10e955-a9a7-4fb3-ac66-4216342c40d2 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.567900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d10e955-a9a7-4fb3-ac66-4216342c40d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.568159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d10e955-a9a7-4fb3-ac66-4216342c40d2 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.568804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d10e955-a9a7-4fb3-ac66-4216342c40d2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:12.570764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d10e955-a9a7-4fb3-ac66-4216342c40d2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:12.662351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 931/1855] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:04:12 2022] POST /identity/v3/projects => generated 375 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:12.663511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca0ff9a-8a3d-4843-973d-82c21b0d462a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:12.674300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca0ff9a-8a3d-4843-973d-82c21b0d462a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:12.743050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf5f1df-cbd8-4530-bdb2-93be6d675417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:12.743655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf5f1df-cbd8-4530-bdb2-93be6d675417 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.743933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf5f1df-cbd8-4530-bdb2-93be6d675417 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.744206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf5f1df-cbd8-4530-bdb2-93be6d675417 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.744849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf5f1df-cbd8-4530-bdb2-93be6d675417 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:12.746830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf5f1df-cbd8-4530-bdb2-93be6d675417 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:12.851744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 925/1856] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:04:12 2022] POST /identity/v3/users => generated 294 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:04:12.855883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d10e955-a9a7-4fb3-ac66-4216342c40d2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:12.866453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d10e955-a9a7-4fb3-ac66-4216342c40d2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:12.870999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add73eab-d3f8-43f8-8fa8-6585f78637ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:12.871592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add73eab-d3f8-43f8-8fa8-6585f78637ef admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.871877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add73eab-d3f8-43f8-8fa8-6585f78637ef admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.872148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add73eab-d3f8-43f8-8fa8-6585f78637ef admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.877027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add73eab-d3f8-43f8-8fa8-6585f78637ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:12.877027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add73eab-d3f8-43f8-8fa8-6585f78637ef admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:12.885318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 932/1857] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:12.889298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf5f1df-cbd8-4530-bdb2-93be6d675417 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:12.899115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf5f1df-cbd8-4530-bdb2-93be6d675417 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:12.904318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c063d8-23fa-4a29-a681-be18f71b668f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:12.904958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c063d8-23fa-4a29-a681-be18f71b668f admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:12.905235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c063d8-23fa-4a29-a681-be18f71b668f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:12.905506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c063d8-23fa-4a29-a681-be18f71b668f admin admin] PATH_INFO: `/v3/projects/bfe37b15897f4ec289debeb7e54d8acd/users/391434e4755348be89b015dc1e4662a1/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:12.919841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c063d8-23fa-4a29-a681-be18f71b668f admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfe37b15897f4ec289debeb7e54d8acd, user_id=391434e4755348be89b015dc1e4662a1, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:12.922609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c063d8-23fa-4a29-a681-be18f71b668f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:12.995067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:13.003064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 926/1858] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:12 2022] PUT /identity/v3/projects/bfe37b15897f4ec289debeb7e54d8acd/users/391434e4755348be89b015dc1e4662a1/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:13.005568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add73eab-d3f8-43f8-8fa8-6585f78637ef admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:13.015754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add73eab-d3f8-43f8-8fa8-6585f78637ef admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:13.043967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:13.081279 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c124fc9-08d2-4844-889f-c5ab0f98f7cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:13.081893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c124fc9-08d2-4844-889f-c5ab0f98f7cb admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:13.083323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c124fc9-08d2-4844-889f-c5ab0f98f7cb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:13.083323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c124fc9-08d2-4844-889f-c5ab0f98f7cb admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:13.083323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c124fc9-08d2-4844-889f-c5ab0f98f7cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:13.084779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c124fc9-08d2-4844-889f-c5ab0f98f7cb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:13.095210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 933/1859] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:13 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:13.099261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c063d8-23fa-4a29-a681-be18f71b668f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:13.110128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c063d8-23fa-4a29-a681-be18f71b668f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:13.114896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:13.115579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:13.115867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:13.116144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] PATH_INFO: `/v3/projects/bfe37b15897f4ec289debeb7e54d8acd/users/391434e4755348be89b015dc1e4662a1/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:13.118075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfe37b15897f4ec289debeb7e54d8acd, user_id=391434e4755348be89b015dc1e4662a1, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:13.120780 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:13.130956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 927/1860] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:13 2022] PUT /identity/v3/projects/bfe37b15897f4ec289debeb7e54d8acd/users/391434e4755348be89b015dc1e4662a1/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:13.142001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63551e1-aea2-4c92-98d8-9b41a84b52ed None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:13.142297 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63551e1-aea2-4c92-98d8-9b41a84b52ed None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:13.142586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63551e1-aea2-4c92-98d8-9b41a84b52ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:13.173988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a63551e1-aea2-4c92-98d8-9b41a84b52ed None None] MFA Rules not processed for user `391434e4755348be89b015dc1e4662a1`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:13.175225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a63551e1-aea2-4c92-98d8-9b41a84b52ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:13.251640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 934/1861] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:13 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:13.262905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:13.264659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:13.323379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e98eaf-8751-49cc-9f1c-ca4cf436e3e8 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:13.327050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f816a4-2eeb-4c9d-9ff7-dc8d7d1f8763 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:13.327645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f816a4-2eeb-4c9d-9ff7-dc8d7d1f8763 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:13.327919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f816a4-2eeb-4c9d-9ff7-dc8d7d1f8763 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:13.328201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f816a4-2eeb-4c9d-9ff7-dc8d7d1f8763 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:13.396094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f816a4-2eeb-4c9d-9ff7-dc8d7d1f8763 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:13.397562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f816a4-2eeb-4c9d-9ff7-dc8d7d1f8763 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:13.407210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 928/1862] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:13 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:13.474682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63551e1-aea2-4c92-98d8-9b41a84b52ed None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:13.476270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a63551e1-aea2-4c92-98d8-9b41a84b52ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:13.547946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63551e1-aea2-4c92-98d8-9b41a84b52ed None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:13.552775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:13.553689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:13.553967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:13.554266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] PATH_INFO: `/v3/users/6f1e7de6e1364136930396fff0444752` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:13.555998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f1e7de6e1364136930396fff0444752)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:13.558703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:13.797776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f1e7de6e1364136930396fff0444752'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:13.797776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:13.858242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f1e7de6e1364136930396fff0444752'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:13.874487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f1e7de6e1364136930396fff0444752'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:13.883264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f1e7de6e1364136930396fff0444752'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:13.884364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f1e7de6e1364136930396fff0444752'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:13.892105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 935/1863] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:04:13 2022] DELETE /identity/v3/users/6f1e7de6e1364136930396fff0444752 => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:13.896741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fa0f23-2f73-47b8-aadc-afb4b3ffcaa2 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:13.897256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fa0f23-2f73-47b8-aadc-afb4b3ffcaa2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:13.897455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fa0f23-2f73-47b8-aadc-afb4b3ffcaa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:13.918008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c2fa0f23-2f73-47b8-aadc-afb4b3ffcaa2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:13.919265 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c2fa0f23-2f73-47b8-aadc-afb4b3ffcaa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:13.979816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 929/1864] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:13.991037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:13.992683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:14.051231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce52f7b0-07bc-4ad9-9ebc-5c10a7922335 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:14.055054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95a6887-7172-4f66-a274-c8443ccb2917 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:14.055695 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95a6887-7172-4f66-a274-c8443ccb2917 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:14.055993 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95a6887-7172-4f66-a274-c8443ccb2917 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:14.056299 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95a6887-7172-4f66-a274-c8443ccb2917 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:14.065676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95a6887-7172-4f66-a274-c8443ccb2917 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:14.067286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95a6887-7172-4f66-a274-c8443ccb2917 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:14.079428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 936/1865] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:14.567294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fa0f23-2f73-47b8-aadc-afb4b3ffcaa2 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:14.568744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c2fa0f23-2f73-47b8-aadc-afb4b3ffcaa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:14.583019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2fa0f23-2f73-47b8-aadc-afb4b3ffcaa2 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:14.587974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:14.589204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:14.589499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:14.589781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] PATH_INFO: `/v3/projects/9fb0ecaa29134499a7f37e48ed59fc72` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:14.591020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fb0ecaa29134499a7f37e48ed59fc72)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:14.593023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:14.723890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:14.732711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fb0ecaa29134499a7f37e48ed59fc72', 'request_id': 'req-a162fd99-394b-448b-9f12-5d497609a1d7'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:14.739082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fb0ecaa29134499a7f37e48ed59fc72', 'request_id': 'req-a162fd99-394b-448b-9f12-5d497609a1d7'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:14.745135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fb0ecaa29134499a7f37e48ed59fc72', 'request_id': 'req-a162fd99-394b-448b-9f12-5d497609a1d7'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:14.807577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] The token cache is being invalidate because project 9fb0ecaa29134499a7f37e48ed59fc72 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:04:14.807817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:14.808514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:14.810018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 930/1866] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:04:14 2022] DELETE /identity/v3/projects/9fb0ecaa29134499a7f37e48ed59fc72 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:14.814064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95a6887-7172-4f66-a274-c8443ccb2917 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:14.824444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95a6887-7172-4f66-a274-c8443ccb2917 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:14.881881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:14.882592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:14.882943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:14.883316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] PATH_INFO: `/v3/users/646561c045bd4b21b9d95bd0e8ebcc45` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:14.884548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] RBAC: Authorizing `identity:delete_user(user_id=646561c045bd4b21b9d95bd0e8ebcc45)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:14.886481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:15.058906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '646561c045bd4b21b9d95bd0e8ebcc45'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.142796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '646561c045bd4b21b9d95bd0e8ebcc45'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.167313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '646561c045bd4b21b9d95bd0e8ebcc45'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.170390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:15.183785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '646561c045bd4b21b9d95bd0e8ebcc45'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.184516 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '646561c045bd4b21b9d95bd0e8ebcc45'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.190840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 937/1867] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:04:14 2022] DELETE /identity/v3/users/646561c045bd4b21b9d95bd0e8ebcc45 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:15.646797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:15.649864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:15.662705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a162fd99-394b-448b-9f12-5d497609a1d7 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:15.727038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:15.727714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:15.727994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:15.728267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] PATH_INFO: `/v3/projects/65c5ed4451c64fa2b15e98e0e13d60b1` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:15.729379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65c5ed4451c64fa2b15e98e0e13d60b1)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:15.731322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:15.830037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65c5ed4451c64fa2b15e98e0e13d60b1', 'request_id': 'req-3e10dac6-6046-4b20-a463-b6d0d686d284'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.836340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65c5ed4451c64fa2b15e98e0e13d60b1', 'request_id': 'req-3e10dac6-6046-4b20-a463-b6d0d686d284'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.842468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65c5ed4451c64fa2b15e98e0e13d60b1', 'request_id': 'req-3e10dac6-6046-4b20-a463-b6d0d686d284'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.907264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] The token cache is being invalidate because project 65c5ed4451c64fa2b15e98e0e13d60b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:04:15.907599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.908382 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:15.909977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 931/1868] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:04:15 2022] DELETE /identity/v3/projects/65c5ed4451c64fa2b15e98e0e13d60b1 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:15.914267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:15.915717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:15.929447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f6a807-f1c8-472d-a9d3-3c125488587c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:15.989685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:15.990357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:15.990641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:15.990944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] PATH_INFO: `/v3/users/086a08be2c3045c49e38de8f315df67a` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:15.992126 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] RBAC: Authorizing `identity:delete_user(user_id=086a08be2c3045c49e38de8f315df67a)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:15.994580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:16.528546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '086a08be2c3045c49e38de8f315df67a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:16.594428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '086a08be2c3045c49e38de8f315df67a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:16.606703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '086a08be2c3045c49e38de8f315df67a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:16.613559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '086a08be2c3045c49e38de8f315df67a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:16.614189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cf9913bd-e24d-4f4f-a680-9e2a6d8dd726 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '086a08be2c3045c49e38de8f315df67a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:16.620081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 938/1869] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:04:15 2022] DELETE /identity/v3/users/086a08be2c3045c49e38de8f315df67a => generated 0 bytes in 706 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:17.059368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:17.060847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:17.080766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e10dac6-6046-4b20-a463-b6d0d686d284 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:17.133134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:17.139986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:17.140625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:17.140907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:17.141184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] PATH_INFO: `/v3/projects/d4067053a5a340338bfbd7b36f90e311` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:17.142315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4067053a5a340338bfbd7b36f90e311)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:17.144330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:17.268179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4067053a5a340338bfbd7b36f90e311', 'request_id': 'req-ee02a551-2e07-4f0a-b83c-afa25112448c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:17.280500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4067053a5a340338bfbd7b36f90e311', 'request_id': 'req-ee02a551-2e07-4f0a-b83c-afa25112448c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:17.286119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4067053a5a340338bfbd7b36f90e311', 'request_id': 'req-ee02a551-2e07-4f0a-b83c-afa25112448c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:17.422073 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] The token cache is being invalidate because project d4067053a5a340338bfbd7b36f90e311 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:04:17.428615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:17.428615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ee02a551-2e07-4f0a-b83c-afa25112448c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:17.428615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 932/1870] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:04:17 2022] DELETE /identity/v3/projects/d4067053a5a340338bfbd7b36f90e311 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:17.433251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e570a23c-7e8f-44ef-ab7c-3cee8b54bca4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:17.433717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e570a23c-7e8f-44ef-ab7c-3cee8b54bca4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:17.433816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e570a23c-7e8f-44ef-ab7c-3cee8b54bca4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:17.458636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e570a23c-7e8f-44ef-ab7c-3cee8b54bca4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:17.460150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e570a23c-7e8f-44ef-ab7c-3cee8b54bca4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:17.546362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 939/1871] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:17.554689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c359eca-925b-49a8-b3f3-ac175007b22d None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:17.554999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c359eca-925b-49a8-b3f3-ac175007b22d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:17.555823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c359eca-925b-49a8-b3f3-ac175007b22d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:17.591457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-4c359eca-925b-49a8-b3f3-ac175007b22d None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:17.592926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4c359eca-925b-49a8-b3f3-ac175007b22d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:17.601946 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 933/1872] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:17.606321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e570a23c-7e8f-44ef-ab7c-3cee8b54bca4 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:17.616189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e570a23c-7e8f-44ef-ab7c-3cee8b54bca4 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:17.621513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:17.622126 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:17.622418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:17.622680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:17.623373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:04:17.623676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:17.625290 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:17.628134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:17.637847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 940/1873] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:04:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:04:17.642451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c359eca-925b-49a8-b3f3-ac175007b22d None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:17.651712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c359eca-925b-49a8-b3f3-ac175007b22d None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:17.656916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9cd33c-87ed-4e09-b6e8-3f4433ca8d9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:17.657507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9cd33c-87ed-4e09-b6e8-3f4433ca8d9c admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:17.657779 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9cd33c-87ed-4e09-b6e8-3f4433ca8d9c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:17.658046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9cd33c-87ed-4e09-b6e8-3f4433ca8d9c admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:17.658674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9cd33c-87ed-4e09-b6e8-3f4433ca8d9c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:17.660600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9cd33c-87ed-4e09-b6e8-3f4433ca8d9c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:17.750662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 934/1874] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:04:17 2022] POST /identity/v3/projects => generated 383 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:17.754887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:17.765678 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105a07ea-071d-44f5-a73d-1fc93b9b1675 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:17.823664 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c918d9-77b0-4cff-97ce-37876c9400a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:17.824171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c918d9-77b0-4cff-97ce-37876c9400a4 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:17.824448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c918d9-77b0-4cff-97ce-37876c9400a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:17.824724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c918d9-77b0-4cff-97ce-37876c9400a4 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:17.825374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c918d9-77b0-4cff-97ce-37876c9400a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:17.827368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c918d9-77b0-4cff-97ce-37876c9400a4 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:17.912841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 941/1875] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:04:17 2022] POST /identity/v3/users => generated 292 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:17.914209 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9cd33c-87ed-4e09-b6e8-3f4433ca8d9c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:17.926578 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9cd33c-87ed-4e09-b6e8-3f4433ca8d9c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:17.932620 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5685cbf5-2eaf-4e7f-86cd-7ca63af70b77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:17.933350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685cbf5-2eaf-4e7f-86cd-7ca63af70b77 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:17.933692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685cbf5-2eaf-4e7f-86cd-7ca63af70b77 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:17.934029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5685cbf5-2eaf-4e7f-86cd-7ca63af70b77 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:17.934795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5685cbf5-2eaf-4e7f-86cd-7ca63af70b77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:17.936551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5685cbf5-2eaf-4e7f-86cd-7ca63af70b77 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:17.946896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 935/1876] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:17 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:17.951155 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c918d9-77b0-4cff-97ce-37876c9400a4 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:17.961988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c918d9-77b0-4cff-97ce-37876c9400a4 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:17.966915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240107f5-437e-44dd-a0c6-1c1ed7ea4f67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:17.976013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240107f5-437e-44dd-a0c6-1c1ed7ea4f67 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:17.976013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240107f5-437e-44dd-a0c6-1c1ed7ea4f67 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:17.976013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240107f5-437e-44dd-a0c6-1c1ed7ea4f67 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:17.976013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240107f5-437e-44dd-a0c6-1c1ed7ea4f67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:17.976013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240107f5-437e-44dd-a0c6-1c1ed7ea4f67 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:17.981694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 942/1877] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:17.985874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5685cbf5-2eaf-4e7f-86cd-7ca63af70b77 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:17.997046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5685cbf5-2eaf-4e7f-86cd-7ca63af70b77 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:18.001827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:18.009384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:18.009384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:18.009384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] PATH_INFO: `/v3/projects/6529fcc0c4c4487ab023942957692c3a/users/86dbc6cb1f144de9b83ccac964a1b3da/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:18.017733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6529fcc0c4c4487ab023942957692c3a, user_id=86dbc6cb1f144de9b83ccac964a1b3da, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:18.020656 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:18.103742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 936/1878] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:17 2022] PUT /identity/v3/projects/6529fcc0c4c4487ab023942957692c3a/users/86dbc6cb1f144de9b83ccac964a1b3da/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:18.108784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696cddb-4f3a-4436-95ff-987d31c31754 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:18.109113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696cddb-4f3a-4436-95ff-987d31c31754 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:18.109392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696cddb-4f3a-4436-95ff-987d31c31754 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:18.129569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3696cddb-4f3a-4436-95ff-987d31c31754 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:18.130783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3696cddb-4f3a-4436-95ff-987d31c31754 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:18.201647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 943/1879] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:18.212216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:18.214266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:18.280727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2b84c7-4a15-427f-86a2-1153ac4db9a4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:18.285500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a7f0d6-57f5-436a-9048-0fc2ae00d397 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:18.286196 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a7f0d6-57f5-436a-9048-0fc2ae00d397 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:18.286524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a7f0d6-57f5-436a-9048-0fc2ae00d397 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:18.286796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a7f0d6-57f5-436a-9048-0fc2ae00d397 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:18.297045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a7f0d6-57f5-436a-9048-0fc2ae00d397 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:18.298466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a7f0d6-57f5-436a-9048-0fc2ae00d397 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:18.308654 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 937/1880] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:18.754933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3696cddb-4f3a-4436-95ff-987d31c31754 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:18.756952 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3696cddb-4f3a-4436-95ff-987d31c31754 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:18.769330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3696cddb-4f3a-4436-95ff-987d31c31754 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:18.775935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:18.776609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:18.776890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:18.777167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] PATH_INFO: `/v3/users/b7fb5f7fce6e48959f13cc17548a9964` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:18.778303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7fb5f7fce6e48959f13cc17548a9964)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:18.780273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:19.010199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7fb5f7fce6e48959f13cc17548a9964'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:19.131434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7fb5f7fce6e48959f13cc17548a9964'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:19.145224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:19.146816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7fb5f7fce6e48959f13cc17548a9964'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:19.154650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7fb5f7fce6e48959f13cc17548a9964'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:19.155427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7fb5f7fce6e48959f13cc17548a9964'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:19.161904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 944/1881] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:04:18 2022] DELETE /identity/v3/users/b7fb5f7fce6e48959f13cc17548a9964 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:19.168446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ad92d9-4275-4aa0-9c7e-b6ad8d47b4ab None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:19.168446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ad92d9-4275-4aa0-9c7e-b6ad8d47b4ab None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:19.168446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ad92d9-4275-4aa0-9c7e-b6ad8d47b4ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:19.189506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-50ad92d9-4275-4aa0-9c7e-b6ad8d47b4ab None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:19.190885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-50ad92d9-4275-4aa0-9c7e-b6ad8d47b4ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:19.220600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:19.264104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 938/1882] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:19.273986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:19.275451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:19.343830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61f57fe-7915-43d8-9139-05f488136f6d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:19.347901 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ef1334-05c6-4008-9f13-1b92cb61c39c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:19.348355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ef1334-05c6-4008-9f13-1b92cb61c39c service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:19.348680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ef1334-05c6-4008-9f13-1b92cb61c39c service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:19.348991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ef1334-05c6-4008-9f13-1b92cb61c39c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:19.362737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ef1334-05c6-4008-9f13-1b92cb61c39c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:19.364414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ef1334-05c6-4008-9f13-1b92cb61c39c service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:19.375085 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 945/1883] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:19.844118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ad92d9-4275-4aa0-9c7e-b6ad8d47b4ab None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:19.845952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-50ad92d9-4275-4aa0-9c7e-b6ad8d47b4ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:19.859437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ad92d9-4275-4aa0-9c7e-b6ad8d47b4ab None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:19.864602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:19.865249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:19.865531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:19.865803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] PATH_INFO: `/v3/projects/a52addd3ae1b4f4dbbd05575db029d62` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:19.867040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] RBAC: Authorizing `identity:delete_project(project_id=a52addd3ae1b4f4dbbd05575db029d62)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:19.868985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:20.011426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a52addd3ae1b4f4dbbd05575db029d62', 'request_id': 'req-dca3565b-6a09-4ffb-9478-a8a054944deb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.018686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a52addd3ae1b4f4dbbd05575db029d62', 'request_id': 'req-dca3565b-6a09-4ffb-9478-a8a054944deb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.025689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a52addd3ae1b4f4dbbd05575db029d62', 'request_id': 'req-dca3565b-6a09-4ffb-9478-a8a054944deb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.093384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] The token cache is being invalidate because project a52addd3ae1b4f4dbbd05575db029d62 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:04:20.093746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.094312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.099088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 939/1884] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:04:19 2022] DELETE /identity/v3/projects/a52addd3ae1b4f4dbbd05575db029d62 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:20.100262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ef1334-05c6-4008-9f13-1b92cb61c39c service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:20.110365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ef1334-05c6-4008-9f13-1b92cb61c39c service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:20.179815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:20.180552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:20.180898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:20.181396 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] PATH_INFO: `/v3/users/391434e4755348be89b015dc1e4662a1` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:20.182621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=391434e4755348be89b015dc1e4662a1)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:20.188640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:20.440862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '391434e4755348be89b015dc1e4662a1'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.515219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '391434e4755348be89b015dc1e4662a1'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.530210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '391434e4755348be89b015dc1e4662a1'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.537319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '391434e4755348be89b015dc1e4662a1'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.538169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-c5fd2cff-d787-46d1-8eb8-504c844717eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '391434e4755348be89b015dc1e4662a1'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:20.544593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 946/1885] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:04:20 2022] DELETE /identity/v3/users/391434e4755348be89b015dc1e4662a1 => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:21.099310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:21.109613 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:21.118624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dca3565b-6a09-4ffb-9478-a8a054944deb admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:21.180942 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:21.181587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:21.181866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:21.182141 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] PATH_INFO: `/v3/projects/bfe37b15897f4ec289debeb7e54d8acd` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:21.183295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bfe37b15897f4ec289debeb7e54d8acd)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:21.185223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:21.296769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bfe37b15897f4ec289debeb7e54d8acd', 'request_id': 'req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:21.302361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:21.304649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bfe37b15897f4ec289debeb7e54d8acd', 'request_id': 'req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:21.311778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bfe37b15897f4ec289debeb7e54d8acd', 'request_id': 'req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:21.367028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] The token cache is being invalidate because project bfe37b15897f4ec289debeb7e54d8acd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:04:21.367349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:21.367951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6c5a6c06-6b19-45aa-856e-5ecd7cb0bc26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:04:21.369566 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 940/1886] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:04:21 2022] DELETE /identity/v3/projects/bfe37b15897f4ec289debeb7e54d8acd => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:21.374726 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b552d0d-5798-46a2-a8f8-91ead8626491 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:21.375231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b552d0d-5798-46a2-a8f8-91ead8626491 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:21.375329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b552d0d-5798-46a2-a8f8-91ead8626491 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:21.398186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-7b552d0d-5798-46a2-a8f8-91ead8626491 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:21.400147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7b552d0d-5798-46a2-a8f8-91ead8626491 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:21.405946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:21.536737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 947/1887] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:21.547884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e814d268-f41e-4a75-b2b8-ed3c9d53320f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:21.548306 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e814d268-f41e-4a75-b2b8-ed3c9d53320f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:21.548660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e814d268-f41e-4a75-b2b8-ed3c9d53320f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:21.581077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e814d268-f41e-4a75-b2b8-ed3c9d53320f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:21.583384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e814d268-f41e-4a75-b2b8-ed3c9d53320f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:21.601995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 941/1888] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:21.604445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b552d0d-5798-46a2-a8f8-91ead8626491 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:21.616773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b552d0d-5798-46a2-a8f8-91ead8626491 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:21.625662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:21.626815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:21.627365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:21.627807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:21.628997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:04:21.629537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:21.634296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:21.651735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 948/1889] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:04:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:04:21.656264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e814d268-f41e-4a75-b2b8-ed3c9d53320f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:21.665784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e814d268-f41e-4a75-b2b8-ed3c9d53320f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:21.671257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e94db-4346-4478-a828-68a09038ed3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:21.671883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e94db-4346-4478-a828-68a09038ed3d admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:21.672166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e94db-4346-4478-a828-68a09038ed3d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:21.672511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963e94db-4346-4478-a828-68a09038ed3d admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:21.673180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e94db-4346-4478-a828-68a09038ed3d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:21.675516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963e94db-4346-4478-a828-68a09038ed3d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:21.773901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 942/1890] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:04:21 2022] POST /identity/v3/projects => generated 373 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:04:21.778801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:21.791587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260c046d-226f-48e0-834e-d528dd7060d5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:21.885683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c70d45c-b177-4a91-ba08-befcdc6da514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:21.885683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70d45c-b177-4a91-ba08-befcdc6da514 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:21.885683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70d45c-b177-4a91-ba08-befcdc6da514 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:21.885683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c70d45c-b177-4a91-ba08-befcdc6da514 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:21.886209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c70d45c-b177-4a91-ba08-befcdc6da514 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:21.886209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c70d45c-b177-4a91-ba08-befcdc6da514 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:21.970322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 949/1891] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:04:21 2022] POST /identity/v3/users => generated 287 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:04:21.975813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e94db-4346-4478-a828-68a09038ed3d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:21.987333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963e94db-4346-4478-a828-68a09038ed3d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:21.992512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde3551-d259-47fd-b257-8468041e6e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:21.993824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde3551-d259-47fd-b257-8468041e6e43 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:21.994111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde3551-d259-47fd-b257-8468041e6e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:21.994385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddde3551-d259-47fd-b257-8468041e6e43 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:21.995244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddde3551-d259-47fd-b257-8468041e6e43 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:21.996787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddde3551-d259-47fd-b257-8468041e6e43 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:22.012430 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 943/1892] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:22.014342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c70d45c-b177-4a91-ba08-befcdc6da514 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:22.029592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c70d45c-b177-4a91-ba08-befcdc6da514 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:22.035635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a60370-528d-40a0-b22b-ebefd34d005f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:22.035986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a60370-528d-40a0-b22b-ebefd34d005f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:22.036343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a60370-528d-40a0-b22b-ebefd34d005f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:22.036692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a60370-528d-40a0-b22b-ebefd34d005f admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:22.038099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a60370-528d-40a0-b22b-ebefd34d005f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:22.039894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a60370-528d-40a0-b22b-ebefd34d005f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:22.065622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 950/1893] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:22 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:04:22.066880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde3551-d259-47fd-b257-8468041e6e43 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:22.080487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddde3551-d259-47fd-b257-8468041e6e43 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:22.100354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:22.102538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:22.107568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:22.107568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] PATH_INFO: `/v3/projects/bbf0f765079a44bc9e68012054763b34/users/e92dd14e2eff4cc0a256708fba70c09d/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:22.148096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bbf0f765079a44bc9e68012054763b34, user_id=e92dd14e2eff4cc0a256708fba70c09d, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:22.152438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:22.243469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 944/1894] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:22 2022] PUT /identity/v3/projects/bbf0f765079a44bc9e68012054763b34/users/e92dd14e2eff4cc0a256708fba70c09d/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:22.249177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da418d3-0eed-440c-a295-7d019511653a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:22.250558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da418d3-0eed-440c-a295-7d019511653a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:22.251415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da418d3-0eed-440c-a295-7d019511653a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:22.287101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-7da418d3-0eed-440c-a295-7d019511653a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:22.288776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7da418d3-0eed-440c-a295-7d019511653a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:22.371131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 951/1895] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:22.372991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:22.377270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:22.440487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69141c9-1c6b-493e-bdea-322ad6901f04 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:22.444729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:22.445339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:22.446273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:22.446550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:22.456260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:22.457696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:22.479664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 945/1896] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:04:24.099140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42148339-5413-4dda-8c87-95f976e051fb None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:24.099735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42148339-5413-4dda-8c87-95f976e051fb None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:24.099844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42148339-5413-4dda-8c87-95f976e051fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:24.142869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-42148339-5413-4dda-8c87-95f976e051fb None None] MFA Rules not processed for user `e92dd14e2eff4cc0a256708fba70c09d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:24.145536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-42148339-5413-4dda-8c87-95f976e051fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:24.155730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:24.274285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 952/1897] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:24 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:24.284531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:24.286187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:24.388796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde1018f-d804-45e0-a775-0cdebdc0a8bf service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:24.394586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:24.395990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:24.395990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:24.396266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:24.406829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:24.408370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:24.432010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 946/1898] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:24.594261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a22fece-72c3-4f18-87ea-288d52d49a96 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:24.594552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a22fece-72c3-4f18-87ea-288d52d49a96 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:24.594831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a22fece-72c3-4f18-87ea-288d52d49a96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:24.629357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0a22fece-72c3-4f18-87ea-288d52d49a96 None None] MFA Rules not processed for user `86dbc6cb1f144de9b83ccac964a1b3da`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:24.630617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0a22fece-72c3-4f18-87ea-288d52d49a96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:24.726733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 953/1899] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:24.727150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:24.730806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:24.780366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705e5b13-7310-4262-9775-c4ed61617b70 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:24.790977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27453844-196a-4298-b9ea-1b85712d28e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:24.791546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27453844-196a-4298-b9ea-1b85712d28e7 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:24.791830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27453844-196a-4298-b9ea-1b85712d28e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:24.792104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27453844-196a-4298-b9ea-1b85712d28e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:24.792751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27453844-196a-4298-b9ea-1b85712d28e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:24.794687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27453844-196a-4298-b9ea-1b85712d28e7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:24.909152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:DELETE=3,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:24.925044 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 947/1900] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:04:24 2022] POST /identity/v3/projects => generated 385 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:24.929197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a22fece-72c3-4f18-87ea-288d52d49a96 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:24.946913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a22fece-72c3-4f18-87ea-288d52d49a96 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:25.022589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bc0bab-ed14-49da-b0d1-0d1ed693261c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:25.023252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bc0bab-ed14-49da-b0d1-0d1ed693261c admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:25.023532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bc0bab-ed14-49da-b0d1-0d1ed693261c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:25.023830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bc0bab-ed14-49da-b0d1-0d1ed693261c admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:25.024489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bc0bab-ed14-49da-b0d1-0d1ed693261c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:25.026482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7bc0bab-ed14-49da-b0d1-0d1ed693261c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:25.108689 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 954/1901] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:04:24 2022] POST /identity/v3/users => generated 299 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:04:25.112712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27453844-196a-4298-b9ea-1b85712d28e7 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:25.124212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27453844-196a-4298-b9ea-1b85712d28e7 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:25.133903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f7c06-293f-4b7f-9b1c-8d3e1e82b11c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:25.133903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f7c06-293f-4b7f-9b1c-8d3e1e82b11c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:25.133903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f7c06-293f-4b7f-9b1c-8d3e1e82b11c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:25.133903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1f7c06-293f-4b7f-9b1c-8d3e1e82b11c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:25.134234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1f7c06-293f-4b7f-9b1c-8d3e1e82b11c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:25.135213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1f7c06-293f-4b7f-9b1c-8d3e1e82b11c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:25.166216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 948/1902] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:25 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:25.170490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bc0bab-ed14-49da-b0d1-0d1ed693261c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:25.182302 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7bc0bab-ed14-49da-b0d1-0d1ed693261c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:25.188128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59886e5-5f66-4aa3-9bf5-a4257f5a8bb0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:25.188993 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59886e5-5f66-4aa3-9bf5-a4257f5a8bb0 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:25.189558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59886e5-5f66-4aa3-9bf5-a4257f5a8bb0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:25.189993 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59886e5-5f66-4aa3-9bf5-a4257f5a8bb0 admin admin] PATH_INFO: `/v3/projects/bfae1b09b3b84a8ba630a380b44b1f8f/users/4599d4912938467692834ae80acf15d5/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:25.208779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59886e5-5f66-4aa3-9bf5-a4257f5a8bb0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfae1b09b3b84a8ba630a380b44b1f8f, user_id=4599d4912938467692834ae80acf15d5, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:25.211812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59886e5-5f66-4aa3-9bf5-a4257f5a8bb0 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:25.315888 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:25.321142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 955/1903] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:25 2022] PUT /identity/v3/projects/bfae1b09b3b84a8ba630a380b44b1f8f/users/4599d4912938467692834ae80acf15d5/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:25.325180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f7c06-293f-4b7f-9b1c-8d3e1e82b11c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:25.339346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1f7c06-293f-4b7f-9b1c-8d3e1e82b11c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:25.466850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc747a13-58d7-4720-953e-64c28ee7aa49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:25.466850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc747a13-58d7-4720-953e-64c28ee7aa49 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:25.466850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc747a13-58d7-4720-953e-64c28ee7aa49 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:25.466850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc747a13-58d7-4720-953e-64c28ee7aa49 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:25.467864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc747a13-58d7-4720-953e-64c28ee7aa49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:25.467864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc747a13-58d7-4720-953e-64c28ee7aa49 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:25.467864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 949/1904] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:04:25 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:04:25.468392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59886e5-5f66-4aa3-9bf5-a4257f5a8bb0 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:25.471187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59886e5-5f66-4aa3-9bf5-a4257f5a8bb0 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:25.476164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ff1926-d61b-4398-bc1a-ddf9dc0825b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:25.477355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff1926-d61b-4398-bc1a-ddf9dc0825b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:25.477648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff1926-d61b-4398-bc1a-ddf9dc0825b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:25.477930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff1926-d61b-4398-bc1a-ddf9dc0825b0 admin admin] PATH_INFO: `/v3/projects/bfae1b09b3b84a8ba630a380b44b1f8f/users/4599d4912938467692834ae80acf15d5/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:25.480592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff1926-d61b-4398-bc1a-ddf9dc0825b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bfae1b09b3b84a8ba630a380b44b1f8f, user_id=4599d4912938467692834ae80acf15d5, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:25.483410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff1926-d61b-4398-bc1a-ddf9dc0825b0 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:25.496264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 956/1905] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:04:25 2022] PUT /identity/v3/projects/bfae1b09b3b84a8ba630a380b44b1f8f/users/4599d4912938467692834ae80acf15d5/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:04:26.351926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f73977-df98-48fc-b807-7ca6db78894d None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:26.352755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f73977-df98-48fc-b807-7ca6db78894d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:26.356637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f73977-df98-48fc-b807-7ca6db78894d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:26.380376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-63f73977-df98-48fc-b807-7ca6db78894d None None] MFA Rules not processed for user `e92dd14e2eff4cc0a256708fba70c09d`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:26.384544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-63f73977-df98-48fc-b807-7ca6db78894d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:26.457094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 950/1906] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:29.851930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c80f98-8d67-4292-b49c-2ab0586cd5b6 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:29.852387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c80f98-8d67-4292-b49c-2ab0586cd5b6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:29.852562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c80f98-8d67-4292-b49c-2ab0586cd5b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:29.880636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a3c80f98-8d67-4292-b49c-2ab0586cd5b6 None None] MFA Rules not processed for user `57b6c2a1a4a248158c5abbabe61ad634`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:29.881881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a3c80f98-8d67-4292-b49c-2ab0586cd5b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:29.966667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 957/1907] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:33.100834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a769f399-d093-4987-8dc4-eb662687f6d5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:33.101278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a769f399-d093-4987-8dc4-eb662687f6d5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:33.101729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a769f399-d093-4987-8dc4-eb662687f6d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:33.130583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a769f399-d093-4987-8dc4-eb662687f6d5 None None] MFA Rules not processed for user `4599d4912938467692834ae80acf15d5`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:33.131886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a769f399-d093-4987-8dc4-eb662687f6d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:33.204669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 951/1908] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:33 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:33.225357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c80f98-8d67-4292-b49c-2ab0586cd5b6 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:33.225357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a3c80f98-8d67-4292-b49c-2ab0586cd5b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:33.304589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c80f98-8d67-4292-b49c-2ab0586cd5b6 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:33.310747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:33.311537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:33.311813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:33.312129 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:33.441929 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:33.443922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:33.466100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 958/1909] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:04:33.824363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1126ccda-5418-4f66-ae19-add53ae9af19 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:33.827220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1126ccda-5418-4f66-ae19-add53ae9af19 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:33.827220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1126ccda-5418-4f66-ae19-add53ae9af19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:33.853777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-1126ccda-5418-4f66-ae19-add53ae9af19 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:33.855032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1126ccda-5418-4f66-ae19-add53ae9af19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:33.865505 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:33.934648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 952/1910] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:04:33 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:33.986432 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:33.988485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:34.042741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:34.066969 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88938062-d386-4d93-a2d2-b2d3cce0c80a service glance] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:34.072739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfd2cb-4b14-4041-8a4f-e1c9f850405f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:34.073353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfd2cb-4b14-4041-8a4f-e1c9f850405f service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:34.073631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfd2cb-4b14-4041-8a4f-e1c9f850405f service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:34.073902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94bfd2cb-4b14-4041-8a4f-e1c9f850405f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:34.087821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfd2cb-4b14-4041-8a4f-e1c9f850405f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:34.089986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94bfd2cb-4b14-4041-8a4f-e1c9f850405f service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:34.116661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 959/1911] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:04:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:04:34.747397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce9ca65-98d2-4d13-9adb-dbaee8e8530a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:34.747894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce9ca65-98d2-4d13-9adb-dbaee8e8530a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:34.747990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce9ca65-98d2-4d13-9adb-dbaee8e8530a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:34.801803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-dce9ca65-98d2-4d13-9adb-dbaee8e8530a None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:34.829384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-dce9ca65-98d2-4d13-9adb-dbaee8e8530a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:34.865883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 953/1912] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:04:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:34.892789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfd2cb-4b14-4041-8a4f-e1c9f850405f service swift] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:34.912101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94bfd2cb-4b14-4041-8a4f-e1c9f850405f service swift] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:34.933280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835fbc6b-ca26-42d3-bb1a-094cad21b31f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:34.935093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835fbc6b-ca26-42d3-bb1a-094cad21b31f service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:34.938635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835fbc6b-ca26-42d3-bb1a-094cad21b31f service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:34.938917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835fbc6b-ca26-42d3-bb1a-094cad21b31f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:34.998012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835fbc6b-ca26-42d3-bb1a-094cad21b31f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:34.998012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835fbc6b-ca26-42d3-bb1a-094cad21b31f service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:35.010130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 960/1913] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:04:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:04:41.541535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0300ea44-ae03-4906-b6f3-a679de73a472 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:41.542049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0300ea44-ae03-4906-b6f3-a679de73a472 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:41.542165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0300ea44-ae03-4906-b6f3-a679de73a472 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:41.619227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-0300ea44-ae03-4906-b6f3-a679de73a472 None None] MFA Rules not processed for user `2c7d8acfab1041fc8edf36fd42b06993`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:41.625366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0300ea44-ae03-4906-b6f3-a679de73a472 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:41.765250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 954/1914] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:41 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:44.989772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=17 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:49.032099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a07ea1d-2c01-49c9-884f-fcc96d8e55a4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:49.032799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a07ea1d-2c01-49c9-884f-fcc96d8e55a4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:49.032799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a07ea1d-2c01-49c9-884f-fcc96d8e55a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:49.091040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3a07ea1d-2c01-49c9-884f-fcc96d8e55a4 None None] MFA Rules not processed for user `7ea5afc217f843389fe051b42616e09d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:49.091040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3a07ea1d-2c01-49c9-884f-fcc96d8e55a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:49.271120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 961/1915] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:51.748863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=47 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:53.037364 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b1523-48de-41b1-ab5b-40e98ffebd7a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:53.038444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b1523-48de-41b1-ab5b-40e98ffebd7a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:53.039051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae4b1523-48de-41b1-ab5b-40e98ffebd7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:53.058689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ae4b1523-48de-41b1-ab5b-40e98ffebd7a None None] MFA Rules not processed for user `86dbc6cb1f144de9b83ccac964a1b3da`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:53.060099 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ae4b1523-48de-41b1-ab5b-40e98ffebd7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:53.068746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 955/1916] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:04:53 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:59.259974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=25 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:04:59.800786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9358077-2031-4d8c-9f26-67b055dd224c None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:59.801110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9358077-2031-4d8c-9f26-67b055dd224c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:59.801387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9358077-2031-4d8c-9f26-67b055dd224c None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:59.802830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 962/1917] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:04:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:04:59.809557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8eff1d-a3b3-433a-af12-66d484cd97b7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:59.810025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8eff1d-a3b3-433a-af12-66d484cd97b7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:59.810119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8eff1d-a3b3-433a-af12-66d484cd97b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:59.842629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-5b8eff1d-a3b3-433a-af12-66d484cd97b7 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:04:59.843880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5b8eff1d-a3b3-433a-af12-66d484cd97b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:59.935112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 956/1918] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:04:59 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:04:59.947188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9358077-2031-4d8c-9f26-67b055dd224c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:04:59.949028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b9358077-2031-4d8c-9f26-67b055dd224c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:04:59.962194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9358077-2031-4d8c-9f26-67b055dd224c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:04:59.965844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642c748e-6495-4ffe-98f8-88b9198d755a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:04:59.966525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c748e-6495-4ffe-98f8-88b9198d755a service glance] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:04:59.966887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c748e-6495-4ffe-98f8-88b9198d755a service glance] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:04:59.967212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642c748e-6495-4ffe-98f8-88b9198d755a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:04:59.978766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642c748e-6495-4ffe-98f8-88b9198d755a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:04:59.980257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-642c748e-6495-4ffe-98f8-88b9198d755a service glance] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:04:59.992790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 963/1919] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:04:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:05:00.177894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5717232f-e2d5-42fc-b32e-d4e2d04cdf1f None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:00.178557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5717232f-e2d5-42fc-b32e-d4e2d04cdf1f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:00.178937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5717232f-e2d5-42fc-b32e-d4e2d04cdf1f None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:00.180809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 957/1920] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:05:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:05:00.189607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a28d8d7-6d22-4f6d-8378-aa9b10da6699 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:00.189816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a28d8d7-6d22-4f6d-8378-aa9b10da6699 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:00.190421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a28d8d7-6d22-4f6d-8378-aa9b10da6699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:00.232514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0a28d8d7-6d22-4f6d-8378-aa9b10da6699 None None] MFA Rules not processed for user `2f705dc9d2e541f1821dda4b5c7b7709`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:05:00.233942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0a28d8d7-6d22-4f6d-8378-aa9b10da6699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:05:00.382595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 964/1921] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:05:00 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:05:00.395421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5717232f-e2d5-42fc-b32e-d4e2d04cdf1f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:05:00.408951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5717232f-e2d5-42fc-b32e-d4e2d04cdf1f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:05:00.414309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea901e85-21af-420b-be34-e07c11696578 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:05:00.415228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea901e85-21af-420b-be34-e07c11696578 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:00.415607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea901e85-21af-420b-be34-e07c11696578 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:00.415985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea901e85-21af-420b-be34-e07c11696578 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:00.424555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea901e85-21af-420b-be34-e07c11696578 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:05:00.426147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea901e85-21af-420b-be34-e07c11696578 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:05:00.436891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 958/1922] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:05:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:05:00.568774 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d0f412-3653-4913-9c90-2bbad677e88f None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:00.568977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d0f412-3653-4913-9c90-2bbad677e88f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:00.569248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d0f412-3653-4913-9c90-2bbad677e88f None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:00.571317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 965/1923] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:05:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 22:05:00.577865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b6bfb-cce3-4ea4-bdca-a7efb87849f8 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:00.578350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b6bfb-cce3-4ea4-bdca-a7efb87849f8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:00.578795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b6bfb-cce3-4ea4-bdca-a7efb87849f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:00.617176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f73b6bfb-cce3-4ea4-bdca-a7efb87849f8 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:05:00.618525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f73b6bfb-cce3-4ea4-bdca-a7efb87849f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:05:00.634927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 959/1924] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:05:00 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:05:00.642511 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d0f412-3653-4913-9c90-2bbad677e88f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:05:00.656399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d0f412-3653-4913-9c90-2bbad677e88f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:05:00.660748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc941b5-c076-4c02-a65d-0dec4ab77740 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:05:00.661305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc941b5-c076-4c02-a65d-0dec4ab77740 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:00.661583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc941b5-c076-4c02-a65d-0dec4ab77740 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:00.661905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc941b5-c076-4c02-a65d-0dec4ab77740 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:00.675580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc941b5-c076-4c02-a65d-0dec4ab77740 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:05:00.677236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc941b5-c076-4c02-a65d-0dec4ab77740 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:05:00.689598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 966/1925] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:05:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:05:01.934601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98507c46-66d5-455c-ba18-8dec5ce7dd7a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:01.935150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98507c46-66d5-455c-ba18-8dec5ce7dd7a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:01.935385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98507c46-66d5-455c-ba18-8dec5ce7dd7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:01.976961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-98507c46-66d5-455c-ba18-8dec5ce7dd7a None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:05:01.978543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-98507c46-66d5-455c-ba18-8dec5ce7dd7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:05:01.988528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 960/1926] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:05:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:05:02.015505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc941b5-c076-4c02-a65d-0dec4ab77740 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:05:02.030583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc941b5-c076-4c02-a65d-0dec4ab77740 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:05:02.036103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcef5726-57b0-491a-b04e-258957e8eb5a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:05:02.038427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef5726-57b0-491a-b04e-258957e8eb5a service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:02.038897 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef5726-57b0-491a-b04e-258957e8eb5a service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:02.039342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcef5726-57b0-491a-b04e-258957e8eb5a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:02.051495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcef5726-57b0-491a-b04e-258957e8eb5a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:05:02.056461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcef5726-57b0-491a-b04e-258957e8eb5a service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:05:02.065047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 967/1927] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:05:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:05:11.953893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=46 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:05:12.060831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=41 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:05:15.434907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98507c46-66d5-455c-ba18-8dec5ce7dd7a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:05:15.436911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-98507c46-66d5-455c-ba18-8dec5ce7dd7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:05:15.619966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98507c46-66d5-455c-ba18-8dec5ce7dd7a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:05:15.633252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af39e59-e413-4ed4-b84a-c89f3ee6f5f9 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:05:15.635048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af39e59-e413-4ed4-b84a-c89f3ee6f5f9 service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:15.635832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af39e59-e413-4ed4-b84a-c89f3ee6f5f9 service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:15.636880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af39e59-e413-4ed4-b84a-c89f3ee6f5f9 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:15.641659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7af39e59-e413-4ed4-b84a-c89f3ee6f5f9 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:05:15.670260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af39e59-e413-4ed4-b84a-c89f3ee6f5f9 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:05:15.673950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af39e59-e413-4ed4-b84a-c89f3ee6f5f9 service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:05:15.702339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 961/1928] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:05:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 268 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:05:25.692534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=32 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:05:26.193193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be86a83-6d83-4e8b-9b5a-33055c0702d9 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:26.193802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be86a83-6d83-4e8b-9b5a-33055c0702d9 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:26.193802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be86a83-6d83-4e8b-9b5a-33055c0702d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:26.215138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6be86a83-6d83-4e8b-9b5a-33055c0702d9 None None] MFA Rules not processed for user `e92dd14e2eff4cc0a256708fba70c09d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:05:26.216390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6be86a83-6d83-4e8b-9b5a-33055c0702d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:05:26.233139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 968/1929] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:05:26 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:05:26.548281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc6b736-4a75-43e2-84d4-d48a58132e1e None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:26.548925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc6b736-4a75-43e2-84d4-d48a58132e1e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:26.549844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc6b736-4a75-43e2-84d4-d48a58132e1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:26.574424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-7cc6b736-4a75-43e2-84d4-d48a58132e1e None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:05:26.576442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7cc6b736-4a75-43e2-84d4-d48a58132e1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:05:26.584846 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 962/1930] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:05:26 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:05:26.597682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be86a83-6d83-4e8b-9b5a-33055c0702d9 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:05:26.620326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be86a83-6d83-4e8b-9b5a-33055c0702d9 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:05:26.631343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a325ab22-77b8-4dee-9b6f-f87737a143aa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:05:26.632081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a325ab22-77b8-4dee-9b6f-f87737a143aa service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:05:26.632454 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a325ab22-77b8-4dee-9b6f-f87737a143aa service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:05:26.633631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a325ab22-77b8-4dee-9b6f-f87737a143aa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:05:26.651648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a325ab22-77b8-4dee-9b6f-f87737a143aa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:05:26.653459 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a325ab22-77b8-4dee-9b6f-f87737a143aa service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:05:26.663197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 969/1931] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:05:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:05:36.565098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:05:36.658798 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=9 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:06:39.370417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc6b736-4a75-43e2-84d4-d48a58132e1e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:06:39.383516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc6b736-4a75-43e2-84d4-d48a58132e1e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:06:39.389657 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b444e66-286a-411a-bb2f-86e59cfa7627 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:06:39.391075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b444e66-286a-411a-bb2f-86e59cfa7627 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:06:39.391511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b444e66-286a-411a-bb2f-86e59cfa7627 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:06:39.392514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b444e66-286a-411a-bb2f-86e59cfa7627 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:06:39.415737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8571162c-f271-4d42-8daa-be8df0697942 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:06:39.416840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8571162c-f271-4d42-8daa-be8df0697942 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:06:39.416964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8571162c-f271-4d42-8daa-be8df0697942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:06:39.420940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b444e66-286a-411a-bb2f-86e59cfa7627 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:06:39.422404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b444e66-286a-411a-bb2f-86e59cfa7627 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:06:39.435230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 963/1932] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:06:39 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:06:39.462150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8571162c-f271-4d42-8daa-be8df0697942 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:06:39.464246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8571162c-f271-4d42-8daa-be8df0697942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:06:39.481563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 970/1933] 10.222.0.20 () {58 vars in 967 bytes} [Wed Oct 19 22:06:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:06:39.546765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b444e66-286a-411a-bb2f-86e59cfa7627 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:06:39.565833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b444e66-286a-411a-bb2f-86e59cfa7627 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:06:39.577257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:06:39.578413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:06:39.578901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:06:39.579298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:06:39.600422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:06:39.603120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:06:39.613423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 964/1934] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:06:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:06:49.450766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:06:49.611198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=12 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:06:59.161080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8571162c-f271-4d42-8daa-be8df0697942 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:06:59.170452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8571162c-f271-4d42-8daa-be8df0697942 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:06:59.174060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:06:59.174655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:06:59.174930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:06:59.175237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:06:59.176944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:06:59.190182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:06:59.191625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:06:59.203018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 971/1935] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:06:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:07:00.390213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:07:00.391633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:07:00.403256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808970b7-0d69-4bf9-9c56-e4a533803241 service swift] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:07:00.406769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:07:00.407391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:07:00.407671 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:07:00.407944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:07:00.409568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:07:00.422840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:07:00.424278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:07:00.435902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 965/1936] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:07:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:07:09.197984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:07:10.430440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:07:22.782400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:07:22.784098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:07:22.802860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564ecc7b-7cb6-4fec-a4de-865c00282d4a service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:07:22.809115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:07:22.809780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:07:22.810054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:07:22.810323 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:07:22.812153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:07:22.868850 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:07:22.870622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:07:22.880251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 972/1937] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:07:22 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:07:32.876047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=28 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:07:41.832132 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:07:41.840569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd717469-05df-400f-9f49-e65cf87ba676 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:07:41.844145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:07:41.844729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:07:41.845003 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:07:41.845272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:07:41.846919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:07:41.856648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:07:41.858060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:07:41.866065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 966/1938] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:07:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:07:50.312415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:07:50.326101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ad953b-a665-41ed-97bc-d72ad63a27bf service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:07:50.336246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:07:50.337458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:07:50.338569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:07:50.339391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:07:50.342183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:07:50.361409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:07:50.367143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:07:50.390719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 973/1939] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:07:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:07:51.862431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:08:00.384647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:08:38.292355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:08:38.375878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d93118a-a2e4-46b0-a6c5-8167ec6ba1d7 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:08:38.379750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:08:38.380564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:08:38.380673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:08:38.380918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:08:38.382674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:08:38.421543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:08:38.423127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:08:38.432565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 967/1940] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:08:38 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:08:38.481076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:08:38.500276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da0ea7d-6772-4580-8918-e945afb70919 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:08:38.503905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:08:38.504496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:08:38.504777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:08:38.505046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:08:38.506765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:08:38.517770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:08:38.519226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:08:38.528088 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 974/1941] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:08:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:08:38.553986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:08:38.562776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751711a2-2c25-4ae2-9f84-f7cd46df7466 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:08:38.566417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:08:38.567026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:08:38.567305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:08:38.567586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:08:38.569259 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:08:38.580396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:08:38.581808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:08:38.590606 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 968/1942] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:08:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:08:46.217282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:08:46.237782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdde1b4-3c73-4812-9807-8275b544c6cb service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:08:46.243681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:08:46.244294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:08:46.244602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:08:46.244879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:08:46.246668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:08:46.309353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:08:46.311120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:08:46.321100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 975/1943] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:08:46 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:08:48.586618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=55 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:08:56.316719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=42 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:09:23.798975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:09:23.800497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:09:23.823299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdc6e0b-c50f-4c6b-8e86-2ebe754ff1e1 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:09:23.826935 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f6653-4014-4c90-920a-652d27a967ed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:09:23.827552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f6653-4014-4c90-920a-652d27a967ed service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:09:23.827828 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f6653-4014-4c90-920a-652d27a967ed service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:09:23.828100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f6653-4014-4c90-920a-652d27a967ed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:09:23.836777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19f6653-4014-4c90-920a-652d27a967ed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:09:23.838191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19f6653-4014-4c90-920a-652d27a967ed service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:09:23.847493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 969/1944] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:09:23 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:09:33.644029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:09:33.655075 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57cb87e7-1eff-4ad3-bc5d-d7e3eb9a8934 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:09:33.660632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9e567a-fede-416b-9855-8c1898a30586 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:09:33.661225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9e567a-fede-416b-9855-8c1898a30586 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:09:33.661504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9e567a-fede-416b-9855-8c1898a30586 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:09:33.661775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9e567a-fede-416b-9855-8c1898a30586 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:09:33.669365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9e567a-fede-416b-9855-8c1898a30586 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:09:33.670944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9e567a-fede-416b-9855-8c1898a30586 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:09:33.680400 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 976/1945] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:09:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:09:33.843223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=11 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:09:43.676185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:10:15.870667 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f6653-4014-4c90-920a-652d27a967ed service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:10:15.892795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f6653-4014-4c90-920a-652d27a967ed service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:10:15.900065 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94336969-c8b2-4f8c-b933-6a0c9f8b97de service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:10:15.900674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94336969-c8b2-4f8c-b933-6a0c9f8b97de service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:10:15.900952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94336969-c8b2-4f8c-b933-6a0c9f8b97de service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:10:15.901229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94336969-c8b2-4f8c-b933-6a0c9f8b97de service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:10:15.910741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94336969-c8b2-4f8c-b933-6a0c9f8b97de service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:10:15.912714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94336969-c8b2-4f8c-b933-6a0c9f8b97de service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:10:15.924448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 970/1946] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:10:15 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:10:25.919561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=10 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:12:22.768304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9e567a-fede-416b-9855-8c1898a30586 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:12:22.796473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9e567a-fede-416b-9855-8c1898a30586 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:12:22.801938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05945e7-bcb8-4238-a719-bfb999a5c133 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:12:22.803103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05945e7-bcb8-4238-a719-bfb999a5c133 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:12:22.803384 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05945e7-bcb8-4238-a719-bfb999a5c133 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:12:22.803655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05945e7-bcb8-4238-a719-bfb999a5c133 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:12:22.811782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05945e7-bcb8-4238-a719-bfb999a5c133 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:12:22.813353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05945e7-bcb8-4238-a719-bfb999a5c133 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:12:22.839727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 977/1947] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:12:22 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:12:32.837019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=20 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:12:37.983565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94336969-c8b2-4f8c-b933-6a0c9f8b97de service placement] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:12:38.002538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94336969-c8b2-4f8c-b933-6a0c9f8b97de service placement] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:12:38.006794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc68e31e-084b-4b34-b3a4-3ee82ac3a0e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:12:38.007408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68e31e-084b-4b34-b3a4-3ee82ac3a0e7 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:12:38.007690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68e31e-084b-4b34-b3a4-3ee82ac3a0e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:12:38.007958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc68e31e-084b-4b34-b3a4-3ee82ac3a0e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:12:38.017809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc68e31e-084b-4b34-b3a4-3ee82ac3a0e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:12:38.019271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc68e31e-084b-4b34-b3a4-3ee82ac3a0e7 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:12:38.029901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 971/1948] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:12:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:12:39.437837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05945e7-bcb8-4238-a719-bfb999a5c133 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:12:39.449791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05945e7-bcb8-4238-a719-bfb999a5c133 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:12:39.456091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:12:39.456814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:12:39.457099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:12:39.457372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:12:39.459318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:12:39.489177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:12:39.491558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:12:39.503677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 978/1949] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:12:39 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:12:40.366922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc68e31e-084b-4b34-b3a4-3ee82ac3a0e7 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:12:40.375956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc68e31e-084b-4b34-b3a4-3ee82ac3a0e7 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:12:40.379873 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91993f0a-f462-41d2-9eba-88661c8fdf3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:12:40.380510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91993f0a-f462-41d2-9eba-88661c8fdf3d service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:12:40.380788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91993f0a-f462-41d2-9eba-88661c8fdf3d service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:12:40.381055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91993f0a-f462-41d2-9eba-88661c8fdf3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:12:40.393174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91993f0a-f462-41d2-9eba-88661c8fdf3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:12:40.394766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91993f0a-f462-41d2-9eba-88661c8fdf3d service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:12:40.408383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 972/1950] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:12:40 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:12:49.498865 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=11 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:12:50.402800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=16 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:13:12.503997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:13:12.515697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c53656d-3950-406c-b1ab-18e18be69279 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:13:12.520034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:13:12.520848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:13:12.520965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:13:12.521232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:13:12.531611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:13:12.533997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:13:12.555518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 979/1951] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:13:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:13:22.539681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:13:37.906665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91993f0a-f462-41d2-9eba-88661c8fdf3d service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:13:37.917316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91993f0a-f462-41d2-9eba-88661c8fdf3d service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:13:37.921217 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a11a337-01be-4b7c-b8a7-4a7cf3bb8148 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:13:37.922010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a11a337-01be-4b7c-b8a7-4a7cf3bb8148 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:13:37.922286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a11a337-01be-4b7c-b8a7-4a7cf3bb8148 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:13:37.922550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a11a337-01be-4b7c-b8a7-4a7cf3bb8148 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:13:37.932674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a11a337-01be-4b7c-b8a7-4a7cf3bb8148 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:13:37.934376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a11a337-01be-4b7c-b8a7-4a7cf3bb8148 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:13:37.963119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 973/1952] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:13:37 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:13:38.704642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:13:38.706083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:13:38.722265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18434b07-b6cb-4a46-952c-825bf4139386 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:13:38.727807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ebea94-f764-4e37-b346-6437c311418b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:13:38.728397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ebea94-f764-4e37-b346-6437c311418b service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:13:38.728677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ebea94-f764-4e37-b346-6437c311418b service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:13:38.728953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ebea94-f764-4e37-b346-6437c311418b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:13:38.736685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ebea94-f764-4e37-b346-6437c311418b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:13:38.738256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ebea94-f764-4e37-b346-6437c311418b service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:13:38.765586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 980/1953] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:13:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:13:39.381684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a11a337-01be-4b7c-b8a7-4a7cf3bb8148 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:13:39.390299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a11a337-01be-4b7c-b8a7-4a7cf3bb8148 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:13:39.393848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92678b9d-0de6-47e7-b097-e3fb654d1701 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:13:39.394460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92678b9d-0de6-47e7-b097-e3fb654d1701 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:13:39.394736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92678b9d-0de6-47e7-b097-e3fb654d1701 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:13:39.395039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92678b9d-0de6-47e7-b097-e3fb654d1701 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:13:39.403733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92678b9d-0de6-47e7-b097-e3fb654d1701 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:13:39.405143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92678b9d-0de6-47e7-b097-e3fb654d1701 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:13:39.432534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 974/1954] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:13:39 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:13:46.361220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ebea94-f764-4e37-b346-6437c311418b service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:13:46.375313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ebea94-f764-4e37-b346-6437c311418b service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:13:46.381529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fcb1c4-96fb-4fec-8e70-77fe18f11681 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:13:46.382678 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fcb1c4-96fb-4fec-8e70-77fe18f11681 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:13:46.382998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fcb1c4-96fb-4fec-8e70-77fe18f11681 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:13:46.383284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fcb1c4-96fb-4fec-8e70-77fe18f11681 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:13:46.393097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fcb1c4-96fb-4fec-8e70-77fe18f11681 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:13:46.394674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fcb1c4-96fb-4fec-8e70-77fe18f11681 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:13:46.426241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 981/1955] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:13:46 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:13:49.429933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=26 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:13:56.423625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=26 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:14:24.469478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92678b9d-0de6-47e7-b097-e3fb654d1701 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:14:24.478637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92678b9d-0de6-47e7-b097-e3fb654d1701 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:14:24.482222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:14:24.482810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:14:24.483129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:14:24.483401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:14:24.485169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:14:24.517787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:14:24.519237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:14:24.536770 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 975/1956] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:14:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:14:33.186912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fcb1c4-96fb-4fec-8e70-77fe18f11681 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:14:33.198877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fcb1c4-96fb-4fec-8e70-77fe18f11681 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:14:33.204477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:14:33.205079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:14:33.205361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:14:33.205671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:14:33.207550 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:14:33.244428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:14:33.246016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:14:33.264670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 982/1957] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:14:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:14:34.534208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=19 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:14:42.933728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:14:42.943263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8121188-87c5-425c-a6ae-a08c2a348e08 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:14:42.946937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:14:42.947823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:14:42.948161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:14:42.948493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:14:42.950423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:14:43.034664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:14:43.036135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:14:43.055332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 976/1958] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:14:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:14:43.262005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=19 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:14:43.569844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:14:43.581709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce0002e-2998-4731-af4b-4da3e21a7ebb service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:14:43.586747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:14:43.587437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:14:43.587715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:14:43.588584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:14:43.590287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:14:43.616626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:14:43.618303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:14:43.628646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 983/1959] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:14:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:14:43.686712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:14:43.688335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:14:43.701431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aeac2a8-7445-4eb2-b9df-55c57738aea7 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:14:43.706102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:14:43.706722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:14:43.707025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:14:43.707296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] PATH_INFO: `/v3/users/e92dd14e2eff4cc0a256708fba70c09d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:14:43.708417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e92dd14e2eff4cc0a256708fba70c09d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:14:43.710314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:14:43.892676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e92dd14e2eff4cc0a256708fba70c09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:43.893338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e92dd14e2eff4cc0a256708fba70c09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:43.905119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e92dd14e2eff4cc0a256708fba70c09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:43.912015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e92dd14e2eff4cc0a256708fba70c09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:43.912712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e92dd14e2eff4cc0a256708fba70c09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:43.992443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 977/1960] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:14:43 2022] DELETE /identity/v3/users/e92dd14e2eff4cc0a256708fba70c09d => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:14:44.229547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:14:44.240266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4236d28-6df6-4eb4-99a7-19a1087a87d5 service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:14:44.310210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:14:44.311510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:14:44.311818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:14:44.312091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] PATH_INFO: `/v3/projects/bbf0f765079a44bc9e68012054763b34` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:14:44.313277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] RBAC: Authorizing `identity:delete_project(project_id=bbf0f765079a44bc9e68012054763b34)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:14:44.315223 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:14:44.460060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bbf0f765079a44bc9e68012054763b34', 'request_id': 'req-b087f575-a16a-4bb3-850d-d7d49b6fb34c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:44.539647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bbf0f765079a44bc9e68012054763b34', 'request_id': 'req-b087f575-a16a-4bb3-850d-d7d49b6fb34c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:44.546591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bbf0f765079a44bc9e68012054763b34', 'request_id': 'req-b087f575-a16a-4bb3-850d-d7d49b6fb34c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:44.558123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] The token cache is being invalidate because project bbf0f765079a44bc9e68012054763b34 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:14:44.558399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:44.561537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:14:44.561537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 984/1961] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:14:44 2022] DELETE /identity/v3/projects/bbf0f765079a44bc9e68012054763b34 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:14:53.988574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=72,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:14:54.555125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=62,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:15:18.834026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:15:18.835459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:15:18.906648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f418c7b-f61d-4bbb-b6b2-4171aeefa5b9 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:15:18.967135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:15:18.967746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:15:18.968033 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:15:18.968301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:15:18.970113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:15:18.982035 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:15:18.983684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:15:18.993365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 978/1962] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:15:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:15:28.989162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=58 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:17:23.178895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:17:23.180334 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:17:23.235599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b087f575-a16a-4bb3-850d-d7d49b6fb34c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:17:23.239141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:17:23.239735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:17:23.240011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:17:23.240277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:17:23.241938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:17:23.309992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:17:23.311473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:17:23.320729 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 985/1963] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:17:23 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:17:33.316459 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=53 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:17:47.348076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:17:47.349670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:17:47.416091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89998d94-8066-43ea-849f-f4c8d45b7cbe service placement] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:17:47.419615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:17:47.420213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:17:47.420490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:17:47.420761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:17:47.422469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:17:47.490885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:17:47.492338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:17:47.503556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 979/1964] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:17:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:17:48.689079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:17:48.690480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:17:48.701977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3536a4fb-bcf2-49f1-a275-46969c740038 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:17:48.706064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:17:48.706656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:17:48.706936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:17:48.707241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:17:48.708903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:17:48.723120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:17:48.724069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:17:48.735287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 986/1965] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:17:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:17:57.498270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=59 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:17:58.730170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:18:18.468874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:18:18.477936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf82cb5-7a1a-416d-aacc-1f114c1f6a7b service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:18:18.481669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:18:18.482216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:18:18.482495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:18:18.482768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:18:18.485079 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:18:18.494537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:18:18.495992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:18:18.504626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 980/1966] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:18:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:18:28.500990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:18:38.541961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:18:38.578886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b6d99-c5b8-4f25-a6ff-401d685989e6 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:18:38.582519 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:18:38.583152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:18:38.583429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:18:38.583703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:18:38.585433 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:18:38.677450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:18:38.678869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:18:38.687887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 987/1967] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:18:38 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:18:39.030507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:18:39.049414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09582520-ace1-4d96-9a59-a352cc4c770d service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:18:39.053054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:18:39.053673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:18:39.053951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:18:39.054222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:18:39.055964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:18:39.127273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:18:39.134408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:18:39.137690 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 981/1968] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:18:39 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:18:39.242545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:18:39.243986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:18:39.259890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bde9e3-5d0e-4751-b7df-620b7c2a172f service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:18:39.265936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:18:39.266526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:18:39.266806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:18:39.267116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:18:39.268791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:18:39.337345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:18:39.338936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:18:39.348056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 988/1969] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:18:39 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:18:45.911296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:18:45.919883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f682b984-4b13-43f0-86da-eb165d9684b9 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:18:45.923484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f86a58a-71b9-418f-b3c6-b09c1a7ac836 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:18:45.924080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f86a58a-71b9-418f-b3c6-b09c1a7ac836 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:18:45.924358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f86a58a-71b9-418f-b3c6-b09c1a7ac836 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:18:45.924631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f86a58a-71b9-418f-b3c6-b09c1a7ac836 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:18:45.926330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3f86a58a-71b9-418f-b3c6-b09c1a7ac836 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:18:46.001902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f86a58a-71b9-418f-b3c6-b09c1a7ac836 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:18:46.003861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f86a58a-71b9-418f-b3c6-b09c1a7ac836 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:18:46.012869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 982/1970] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:18:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:18:49.343790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=88 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:18:56.008676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=71 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:19:32.908125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:19:32.918267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850fc02c-b90a-48c9-8347-d4a0c92bf7f4 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:19:32.922062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec102ab9-5c08-4554-8193-7630a8aca881 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:19:32.922770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec102ab9-5c08-4554-8193-7630a8aca881 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:19:32.923045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec102ab9-5c08-4554-8193-7630a8aca881 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:19:32.923315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec102ab9-5c08-4554-8193-7630a8aca881 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:19:32.925159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ec102ab9-5c08-4554-8193-7630a8aca881 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:19:32.987447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec102ab9-5c08-4554-8193-7630a8aca881 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:19:32.988805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec102ab9-5c08-4554-8193-7630a8aca881 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:19:32.997642 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 989/1971] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:19:32 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:19:42.993547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=28 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:20:23.846874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd677682-5168-4325-81a0-9917f525ec8d None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:20:23.847400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd677682-5168-4325-81a0-9917f525ec8d None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:20:23.847516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd677682-5168-4325-81a0-9917f525ec8d None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:20:23.849570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 983/1972] 10.222.0.20 () {54 vars in 961 bytes} [Wed Oct 19 22:20:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:20:23.854856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856995bb-bc39-46f3-a766-73ae86c81851 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:20:23.855372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856995bb-bc39-46f3-a766-73ae86c81851 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:20:23.855472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856995bb-bc39-46f3-a766-73ae86c81851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:20:23.917374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-856995bb-bc39-46f3-a766-73ae86c81851 None None] MFA Rules not processed for user `0a6d53677cb64aaba09e89eb527879cb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:20:23.918635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-856995bb-bc39-46f3-a766-73ae86c81851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:20:23.950565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 990/1973] 10.222.0.20 () {58 vars in 1058 bytes} [Wed Oct 19 22:20:23 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:20:23.958686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd677682-5168-4325-81a0-9917f525ec8d None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:20:23.971521 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd677682-5168-4325-81a0-9917f525ec8d None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:20:23.976856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e176c7f-f4af-4374-a5c8-21e5bd85a148 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:20:23.977437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e176c7f-f4af-4374-a5c8-21e5bd85a148 service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:20:23.977712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e176c7f-f4af-4374-a5c8-21e5bd85a148 service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:20:23.977982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e176c7f-f4af-4374-a5c8-21e5bd85a148 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:20:23.986086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e176c7f-f4af-4374-a5c8-21e5bd85a148 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:20:23.987675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e176c7f-f4af-4374-a5c8-21e5bd85a148 service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:20:23.997724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 984/1974] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:20:23 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:20:33.943006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=24 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:20:33.993652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:20:41.645886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856995bb-bc39-46f3-a766-73ae86c81851 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:20:41.661922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856995bb-bc39-46f3-a766-73ae86c81851 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:20:41.668435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:20:41.669172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:20:41.669560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:20:41.670015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:20:41.671839 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:20:41.685976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:20:41.687803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:20:41.701348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 991/1975] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:20:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:20:51.695068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:22:23.151749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e176c7f-f4af-4374-a5c8-21e5bd85a148 service placement] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:22:23.181133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e176c7f-f4af-4374-a5c8-21e5bd85a148 service placement] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:22:23.186552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75629be9-0043-4eef-882d-ea9afed20d64 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:22:23.187171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75629be9-0043-4eef-882d-ea9afed20d64 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:22:23.187449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75629be9-0043-4eef-882d-ea9afed20d64 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:22:23.187718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75629be9-0043-4eef-882d-ea9afed20d64 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:22:23.195775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75629be9-0043-4eef-882d-ea9afed20d64 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:22:23.197345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75629be9-0043-4eef-882d-ea9afed20d64 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:22:23.223129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 985/1976] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:22:23 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 72 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:22:33.220556 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=20 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:22:46.901522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:22:46.921393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de477a17-8e44-45e6-8b21-fcf9d514eb7d service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:22:46.925130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9112963e-3be0-4d51-b2e0-c3f34016405f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:22:46.925730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9112963e-3be0-4d51-b2e0-c3f34016405f service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:22:46.926006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9112963e-3be0-4d51-b2e0-c3f34016405f service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:22:46.926272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9112963e-3be0-4d51-b2e0-c3f34016405f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:22:46.937232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9112963e-3be0-4d51-b2e0-c3f34016405f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:22:46.938665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9112963e-3be0-4d51-b2e0-c3f34016405f service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:22:46.949940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 992/1977] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:22:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:22:49.276457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75629be9-0043-4eef-882d-ea9afed20d64 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:22:49.286162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75629be9-0043-4eef-882d-ea9afed20d64 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:22:49.290454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac06d44-dc62-4d8b-92a7-73ab38e96ee1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:22:49.291227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac06d44-dc62-4d8b-92a7-73ab38e96ee1 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:22:49.291506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac06d44-dc62-4d8b-92a7-73ab38e96ee1 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:22:49.291773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac06d44-dc62-4d8b-92a7-73ab38e96ee1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:22:49.302307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac06d44-dc62-4d8b-92a7-73ab38e96ee1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:22:49.303762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac06d44-dc62-4d8b-92a7-73ab38e96ee1 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:22:49.315366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 986/1978] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:22:49 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:22:56.945206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=10 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:22:59.309781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:23:21.510630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9112963e-3be0-4d51-b2e0-c3f34016405f service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:23:21.520333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9112963e-3be0-4d51-b2e0-c3f34016405f service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:23:21.524881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:23:21.525625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:23:21.525911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:23:21.526183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:23:21.533910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:23:21.535416 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:23:21.544438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 993/1979] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:23:21 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:23:31.540243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:23:38.133244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac06d44-dc62-4d8b-92a7-73ab38e96ee1 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:23:38.141836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac06d44-dc62-4d8b-92a7-73ab38e96ee1 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:23:38.145502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75b9068-1774-4f29-a75d-4049cc6e8adb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:23:38.146095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75b9068-1774-4f29-a75d-4049cc6e8adb service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:23:38.146375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75b9068-1774-4f29-a75d-4049cc6e8adb service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:23:38.146647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75b9068-1774-4f29-a75d-4049cc6e8adb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:23:38.155002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75b9068-1774-4f29-a75d-4049cc6e8adb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:23:38.156418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75b9068-1774-4f29-a75d-4049cc6e8adb service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:23:38.165686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 987/1980] 10.222.0.20 () {60 vars in 1396 bytes} [Wed Oct 19 22:23:38 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3260 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:23:38.681868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:23:38.683449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:23:38.712948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12b57f0-676b-4e6b-ae83-b0ea8607cf8a service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:23:38.718413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d10cac0-1c6f-49a7-98dd-ea30c35717f9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:23:38.719033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d10cac0-1c6f-49a7-98dd-ea30c35717f9 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:23:38.719315 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d10cac0-1c6f-49a7-98dd-ea30c35717f9 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:23:38.719587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d10cac0-1c6f-49a7-98dd-ea30c35717f9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:23:38.728038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d10cac0-1c6f-49a7-98dd-ea30c35717f9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:23:38.729616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d10cac0-1c6f-49a7-98dd-ea30c35717f9 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:23:38.738480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 994/1981] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:23:38 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:23:46.219916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75b9068-1774-4f29-a75d-4049cc6e8adb service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:23:46.233445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75b9068-1774-4f29-a75d-4049cc6e8adb service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:23:46.239060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:23:46.239674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:23:46.239954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:23:46.240228 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:23:46.248662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:23:46.250248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:23:46.260304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 988/1982] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:23:46 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:23:48.734338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=11 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:23:56.255182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=12 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:24:33.270178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d10cac0-1c6f-49a7-98dd-ea30c35717f9 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:24:33.281956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d10cac0-1c6f-49a7-98dd-ea30c35717f9 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:24:33.287360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:24:33.287946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:24:33.288225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:24:33.289154 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:24:33.296778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:24:33.298342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:24:33.324567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 995/1983] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:24:33 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:24:41.650636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:24:41.650636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:24:41.678068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be9cc2a-a4a8-42c8-b34d-6d116052eb8b service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:24:41.682412 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f4b9cf-1eaa-4d4c-bfe1-ed901ed582c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:24:41.693479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f4b9cf-1eaa-4d4c-bfe1-ed901ed582c5 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:24:41.693479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f4b9cf-1eaa-4d4c-bfe1-ed901ed582c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:24:41.693479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f4b9cf-1eaa-4d4c-bfe1-ed901ed582c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:24:41.693479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f4b9cf-1eaa-4d4c-bfe1-ed901ed582c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:24:41.695858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16f4b9cf-1eaa-4d4c-bfe1-ed901ed582c5 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:24:41.725435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 989/1984] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:24:41 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:24:43.321874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=13 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:24:51.722345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=18 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:25:31.828601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:25:31.829996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:25:31.920613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1efd9b-874f-47b8-b1a6-cd8da46ad7e8 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:25:31.926348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:25:31.926983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:25:31.927265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:25:31.927537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:25:31.929413 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:25:31.941772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:25:31.943422 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:25:31.953909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 996/1985] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:25:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:25:41.950081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=37 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:26:40.514572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe91b3-ece0-4bee-a295-d4b9498be018 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:40.515060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe91b3-ece0-4bee-a295-d4b9498be018 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:40.515294 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afe91b3-ece0-4bee-a295-d4b9498be018 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:40.572198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-7afe91b3-ece0-4bee-a295-d4b9498be018 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:26:40.573354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7afe91b3-ece0-4bee-a295-d4b9498be018 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:40.641926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 990/1986] 10.222.0.20 () {58 vars in 967 bytes} [Wed Oct 19 22:26:40 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:26:40.871488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:40.872911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:40.967577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a033528-acbb-42fb-a5ec-f066994fd49b service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:40.972741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37270f3-3fcd-4039-bf77-8e5af6f1684a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:40.973348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37270f3-3fcd-4039-bf77-8e5af6f1684a service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:40.973625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37270f3-3fcd-4039-bf77-8e5af6f1684a service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:40.973916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37270f3-3fcd-4039-bf77-8e5af6f1684a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:40.983117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37270f3-3fcd-4039-bf77-8e5af6f1684a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:40.984788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37270f3-3fcd-4039-bf77-8e5af6f1684a service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:40.993164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 997/1987] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:26:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:26:41.626240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe91b3-ece0-4bee-a295-d4b9498be018 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:41.653880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afe91b3-ece0-4bee-a295-d4b9498be018 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:41.653880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:41.654159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:41.654594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:41.654881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:41.659265 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:41.674807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:41.682177 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:41.692012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 991/1988] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:26:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:26:42.507332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37270f3-3fcd-4039-bf77-8e5af6f1684a service swift] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:42.524318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37270f3-3fcd-4039-bf77-8e5af6f1684a service swift] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:42.533093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c76325-c50c-4f86-ad30-abd162142973 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:42.539389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c76325-c50c-4f86-ad30-abd162142973 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:42.539389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c76325-c50c-4f86-ad30-abd162142973 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:42.539389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c76325-c50c-4f86-ad30-abd162142973 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:42.539389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-16c76325-c50c-4f86-ad30-abd162142973 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:42.677418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c76325-c50c-4f86-ad30-abd162142973 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:42.678943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c76325-c50c-4f86-ad30-abd162142973 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:42.704235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 998/1989] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:26:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:26:44.474855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:44.506189 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556fe1ce-eeb7-4d07-8c37-000bef3dec72 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:44.520023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a49641-75ef-4ea7-b859-5566e5f28176 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:44.521717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a49641-75ef-4ea7-b859-5566e5f28176 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:44.523421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a49641-75ef-4ea7-b859-5566e5f28176 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:44.523844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a49641-75ef-4ea7-b859-5566e5f28176 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:44.538524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a49641-75ef-4ea7-b859-5566e5f28176 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:44.541021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a49641-75ef-4ea7-b859-5566e5f28176 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:44.570502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 992/1990] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:26:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:26:44.644541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc19844-09fe-44c7-b50b-b7bf7db91b7b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:44.645740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc19844-09fe-44c7-b50b-b7bf7db91b7b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:44.647584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc19844-09fe-44c7-b50b-b7bf7db91b7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:44.789620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-2cc19844-09fe-44c7-b50b-b7bf7db91b7b None None] MFA Rules not processed for user `5ae13baf5df84b588fa65e8dffec4bbf`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:26:44.793586 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2cc19844-09fe-44c7-b50b-b7bf7db91b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:44.827984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 999/1991] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:26:44 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:26:45.352452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a49641-75ef-4ea7-b859-5566e5f28176 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:45.365569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a49641-75ef-4ea7-b859-5566e5f28176 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:45.372979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:45.373744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:45.375084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:45.375618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:45.378643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:45.396737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:45.413895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:45.437189 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 993/1992] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:26:45 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:26:45.997692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2aa40-6325-413d-b3fd-4d4a4d547824 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:45.998249 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2aa40-6325-413d-b3fd-4d4a4d547824 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:45.998417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f2aa40-6325-413d-b3fd-4d4a4d547824 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:46.013148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:26:46.079234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c1f2aa40-6325-413d-b3fd-4d4a4d547824 None None] MFA Rules not processed for user `f9b4be7642f3441ab83cf81e1029e017`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:26:46.083170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c1f2aa40-6325-413d-b3fd-4d4a4d547824 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:46.104000 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1000/1993] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:26:45 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:26:46.457827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:46.474902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f894b3f7-b052-4975-875a-78debdc08fe4 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:46.481915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f23916-af84-4736-9f02-953ef0fcba95 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:46.482788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f23916-af84-4736-9f02-953ef0fcba95 service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:46.483271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f23916-af84-4736-9f02-953ef0fcba95 service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:46.483736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f23916-af84-4736-9f02-953ef0fcba95 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:46.485808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-72f23916-af84-4736-9f02-953ef0fcba95 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:46.521808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f23916-af84-4736-9f02-953ef0fcba95 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:46.525695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f23916-af84-4736-9f02-953ef0fcba95 service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:46.571232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 994/1994] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:26:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:26:46.876018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f2aa40-6325-413d-b3fd-4d4a4d547824 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:46.886339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f2aa40-6325-413d-b3fd-4d4a4d547824 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:46.890316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aa2f74-6b60-43ff-8494-ada68ee151a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:46.890913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa2f74-6b60-43ff-8494-ada68ee151a4 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:46.891339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa2f74-6b60-43ff-8494-ada68ee151a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:46.891611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aa2f74-6b60-43ff-8494-ada68ee151a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:46.894166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-13aa2f74-6b60-43ff-8494-ada68ee151a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:46.906902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aa2f74-6b60-43ff-8494-ada68ee151a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:46.908374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aa2f74-6b60-43ff-8494-ada68ee151a4 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:46.920905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1001/1995] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:26:46 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:26:47.244753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43084dba-9434-4e68-8427-20953145617c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:47.245283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43084dba-9434-4e68-8427-20953145617c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:47.245763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43084dba-9434-4e68-8427-20953145617c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:47.316552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-43084dba-9434-4e68-8427-20953145617c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:26:47.321324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-43084dba-9434-4e68-8427-20953145617c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:47.331719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 995/1996] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:26:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:26:47.340307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbeb281-5b51-429d-a2bf-5841f43011fa None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:47.341016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbeb281-5b51-429d-a2bf-5841f43011fa None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:47.341303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbeb281-5b51-429d-a2bf-5841f43011fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:47.380767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1bbeb281-5b51-429d-a2bf-5841f43011fa None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:26:47.382194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1bbeb281-5b51-429d-a2bf-5841f43011fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:47.402585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1002/1997] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:26:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:26:47.406650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43084dba-9434-4e68-8427-20953145617c None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:47.417598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43084dba-9434-4e68-8427-20953145617c None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:47.422821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:47.423543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:47.423936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:47.424285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:47.425019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:26:47.425862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:47.427715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:47.441137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 996/1998] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:26:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:26:47.445649 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbeb281-5b51-429d-a2bf-5841f43011fa None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:47.456786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbeb281-5b51-429d-a2bf-5841f43011fa None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:47.460912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199f7291-e065-479d-b423-2b31deb6c151 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:47.461615 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f7291-e065-479d-b423-2b31deb6c151 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:47.461923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f7291-e065-479d-b423-2b31deb6c151 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:47.462202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199f7291-e065-479d-b423-2b31deb6c151 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:47.462873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199f7291-e065-479d-b423-2b31deb6c151 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:47.465047 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199f7291-e065-479d-b423-2b31deb6c151 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:47.573836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1003/1999] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:26:47 2022] POST /identity/v3/projects => generated 371 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:26:47.587347 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:47.595231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2402f758-dc52-44fc-b0d6-d961dfdc1b6a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:47.688435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:26:47.738907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e623fded-377d-4bd4-b427-7c602bb35adb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:47.739813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623fded-377d-4bd4-b427-7c602bb35adb admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:47.740520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623fded-377d-4bd4-b427-7c602bb35adb admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:47.741009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e623fded-377d-4bd4-b427-7c602bb35adb admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:47.741976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e623fded-377d-4bd4-b427-7c602bb35adb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:47.745378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e623fded-377d-4bd4-b427-7c602bb35adb admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:47.873106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 997/2000] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:26:47 2022] POST /identity/v3/users => generated 286 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:26:47.874353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199f7291-e065-479d-b423-2b31deb6c151 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:47.884876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199f7291-e065-479d-b423-2b31deb6c151 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:47.889832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179e2296-17e8-40b4-b295-d9bcdde2edb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:47.890464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e2296-17e8-40b4-b295-d9bcdde2edb8 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:47.890812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e2296-17e8-40b4-b295-d9bcdde2edb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:47.891186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179e2296-17e8-40b4-b295-d9bcdde2edb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:47.891936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179e2296-17e8-40b4-b295-d9bcdde2edb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:47.893543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179e2296-17e8-40b4-b295-d9bcdde2edb8 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:47.907176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1004/2001] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:26:47.911328 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e623fded-377d-4bd4-b427-7c602bb35adb admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:47.922101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e623fded-377d-4bd4-b427-7c602bb35adb admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:47.928955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f93a2bc-3e86-4d8d-930f-7d7494d59f99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:47.929576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f93a2bc-3e86-4d8d-930f-7d7494d59f99 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:47.929857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f93a2bc-3e86-4d8d-930f-7d7494d59f99 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:47.930126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f93a2bc-3e86-4d8d-930f-7d7494d59f99 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:47.931561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f93a2bc-3e86-4d8d-930f-7d7494d59f99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:47.933164 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f93a2bc-3e86-4d8d-930f-7d7494d59f99 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:47.947430 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 998/2002] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:26:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:26:47.948618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179e2296-17e8-40b4-b295-d9bcdde2edb8 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:47.959553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179e2296-17e8-40b4-b295-d9bcdde2edb8 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:47.964613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30465956-3af4-443f-bf5a-1d5c8ccb2d62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:47.965277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30465956-3af4-443f-bf5a-1d5c8ccb2d62 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:47.965554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30465956-3af4-443f-bf5a-1d5c8ccb2d62 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:47.965835 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30465956-3af4-443f-bf5a-1d5c8ccb2d62 admin admin] PATH_INFO: `/v3/projects/0b5ba219e7b44b228ed013f830a61572/users/11c2166928f8414f8ae545f7bbd9fd6f/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:47.982663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30465956-3af4-443f-bf5a-1d5c8ccb2d62 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0b5ba219e7b44b228ed013f830a61572, user_id=11c2166928f8414f8ae545f7bbd9fd6f, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:47.985466 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30465956-3af4-443f-bf5a-1d5c8ccb2d62 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:48.041793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1005/2003] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:26:47 2022] PUT /identity/v3/projects/0b5ba219e7b44b228ed013f830a61572/users/11c2166928f8414f8ae545f7bbd9fd6f/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:26:48.048055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425aff9-da42-4a05-b266-d261cf499b7c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:48.048415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425aff9-da42-4a05-b266-d261cf499b7c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:48.048975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6425aff9-da42-4a05-b266-d261cf499b7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:48.073620 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6425aff9-da42-4a05-b266-d261cf499b7c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:26:48.075271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6425aff9-da42-4a05-b266-d261cf499b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:48.151095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 999/2004] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:26:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:26:48.162991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30465956-3af4-443f-bf5a-1d5c8ccb2d62 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:48.253838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30465956-3af4-443f-bf5a-1d5c8ccb2d62 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:48.259254 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a21546-ae4d-4982-8995-352d5ce39f0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:48.260816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a21546-ae4d-4982-8995-352d5ce39f0d service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:48.261365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a21546-ae4d-4982-8995-352d5ce39f0d service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:48.261738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a21546-ae4d-4982-8995-352d5ce39f0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:48.275193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a21546-ae4d-4982-8995-352d5ce39f0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:48.277163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a21546-ae4d-4982-8995-352d5ce39f0d service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:48.289546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1006/2005] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:26:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:26:49.210799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935708f9-8179-48c2-b4c3-12a33f11fda2 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:49.212057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935708f9-8179-48c2-b4c3-12a33f11fda2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:49.212529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935708f9-8179-48c2-b4c3-12a33f11fda2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:49.266208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-935708f9-8179-48c2-b4c3-12a33f11fda2 None None] MFA Rules not processed for user `57b6c2a1a4a248158c5abbabe61ad634`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:26:49.267695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-935708f9-8179-48c2-b4c3-12a33f11fda2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:49.336763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1000/2006] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:26:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:26:51.013924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a21546-ae4d-4982-8995-352d5ce39f0d service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:51.023118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a21546-ae4d-4982-8995-352d5ce39f0d service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:51.028239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:51.028838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:51.029113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:51.031278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:51.032262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:51.047136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:51.048412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:51.073150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1007/2007] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:26:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:26:56.927123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1665936-941b-46e7-b8f4-2f9b9664c2b6 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:56.928434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1665936-941b-46e7-b8f4-2f9b9664c2b6 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:56.928861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1665936-941b-46e7-b8f4-2f9b9664c2b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:56.986418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d1665936-941b-46e7-b8f4-2f9b9664c2b6 None None] MFA Rules not processed for user `11c2166928f8414f8ae545f7bbd9fd6f`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:26:56.993647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d1665936-941b-46e7-b8f4-2f9b9664c2b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:26:57.065480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:26:57.138188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1001/2008] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:26:56 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:26:57.139416 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:57.151790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a596c4f3-ce82-4f21-9380-2846ae9369df service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:57.156877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fbf7e4-7358-400b-9dc7-4583abfb1c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:57.157669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fbf7e4-7358-400b-9dc7-4583abfb1c0e admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:57.158123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fbf7e4-7358-400b-9dc7-4583abfb1c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:57.158580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fbf7e4-7358-400b-9dc7-4583abfb1c0e admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:57.159437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fbf7e4-7358-400b-9dc7-4583abfb1c0e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:57.161533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fbf7e4-7358-400b-9dc7-4583abfb1c0e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:57.255763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1008/2009] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:26:57 2022] POST /identity/v3/projects => generated 371 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:26:57.258612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1665936-941b-46e7-b8f4-2f9b9664c2b6 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:57.274820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1665936-941b-46e7-b8f4-2f9b9664c2b6 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:57.365960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feead453-8ad9-4713-8235-b691469c1881 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:57.367342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feead453-8ad9-4713-8235-b691469c1881 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:57.367342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feead453-8ad9-4713-8235-b691469c1881 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:57.367618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feead453-8ad9-4713-8235-b691469c1881 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:57.369322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feead453-8ad9-4713-8235-b691469c1881 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:57.372318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feead453-8ad9-4713-8235-b691469c1881 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:57.510729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1002/2010] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:26:57 2022] POST /identity/v3/users => generated 292 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:26:57.510814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fbf7e4-7358-400b-9dc7-4583abfb1c0e admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:57.524950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fbf7e4-7358-400b-9dc7-4583abfb1c0e admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:57.534809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ba2388-1ce0-4a0c-98aa-7db00ac68da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:57.536221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba2388-1ce0-4a0c-98aa-7db00ac68da2 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:57.536653 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba2388-1ce0-4a0c-98aa-7db00ac68da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:57.538333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12ba2388-1ce0-4a0c-98aa-7db00ac68da2 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:57.539197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ba2388-1ce0-4a0c-98aa-7db00ac68da2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:57.541701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12ba2388-1ce0-4a0c-98aa-7db00ac68da2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:57.573356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1009/2011] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:26:57 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:26:57.574497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feead453-8ad9-4713-8235-b691469c1881 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:57.589376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feead453-8ad9-4713-8235-b691469c1881 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:57.594447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdcadfb-896a-401b-923a-5b9fd87ecdf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:57.595252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdcadfb-896a-401b-923a-5b9fd87ecdf4 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:57.596090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdcadfb-896a-401b-923a-5b9fd87ecdf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:57.596090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bdcadfb-896a-401b-923a-5b9fd87ecdf4 admin admin] PATH_INFO: `/v3/projects/2c3ddfd7078c4621b2c42181adcbaf19/users/84c32bcd9bf94ae2b29548d4411884cd/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:57.636626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdcadfb-896a-401b-923a-5b9fd87ecdf4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c3ddfd7078c4621b2c42181adcbaf19, user_id=84c32bcd9bf94ae2b29548d4411884cd, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:57.641356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bdcadfb-896a-401b-923a-5b9fd87ecdf4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:57.757523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1003/2012] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:26:57 2022] PUT /identity/v3/projects/2c3ddfd7078c4621b2c42181adcbaf19/users/84c32bcd9bf94ae2b29548d4411884cd/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:26:57.759367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ba2388-1ce0-4a0c-98aa-7db00ac68da2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:57.778468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12ba2388-1ce0-4a0c-98aa-7db00ac68da2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:57.829958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:26:57.885614 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da410a5b-d505-44f5-9abe-1fe246de2714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:57.886731 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da410a5b-d505-44f5-9abe-1fe246de2714 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:57.887348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da410a5b-d505-44f5-9abe-1fe246de2714 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:57.887848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da410a5b-d505-44f5-9abe-1fe246de2714 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:57.889698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da410a5b-d505-44f5-9abe-1fe246de2714 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:57.891802 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da410a5b-d505-44f5-9abe-1fe246de2714 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:57.910972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1010/2013] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:26:57 2022] GET /identity/v3/roles => generated 1420 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:26:57.911782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdcadfb-896a-401b-923a-5b9fd87ecdf4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:26:57.925804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bdcadfb-896a-401b-923a-5b9fd87ecdf4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:26:57.933348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:26:57.935143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:26:57.935649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:26:57.936133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] PATH_INFO: `/v3/projects/2c3ddfd7078c4621b2c42181adcbaf19/users/84c32bcd9bf94ae2b29548d4411884cd/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:26:57.938891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] RBAC: Authorizing `identity:create_grant(project_id=2c3ddfd7078c4621b2c42181adcbaf19, user_id=84c32bcd9bf94ae2b29548d4411884cd, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:26:57.942307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:26:57.960068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1004/2014] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:26:57 2022] PUT /identity/v3/projects/2c3ddfd7078c4621b2c42181adcbaf19/users/84c32bcd9bf94ae2b29548d4411884cd/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:27:00.173477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da410a5b-d505-44f5-9abe-1fe246de2714 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:00.298797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da410a5b-d505-44f5-9abe-1fe246de2714 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:00.342922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:00.347581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:00.410398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:00.411621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:00.411621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:00.411891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:00.415496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:00.440246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:00.442115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:00.455073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1011/2015] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:27:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 282 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:27:00.485945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-321da621-6e19-45b6-8617-d178da6da9da admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:00.496769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:00.497429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:00.497728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:00.498148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] PATH_INFO: `/v3/users/2c7d8acfab1041fc8edf36fd42b06993` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:00.499542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c7d8acfab1041fc8edf36fd42b06993)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:00.501981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:00.528709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:27:00.731610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c7d8acfab1041fc8edf36fd42b06993'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:00.733486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c7d8acfab1041fc8edf36fd42b06993'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:00.752438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c7d8acfab1041fc8edf36fd42b06993'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:00.761015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c7d8acfab1041fc8edf36fd42b06993'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:00.768702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c7d8acfab1041fc8edf36fd42b06993'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:00.857702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1005/2016] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:27:00 2022] DELETE /identity/v3/users/2c7d8acfab1041fc8edf36fd42b06993 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:27:01.230685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:01.236310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:01.238104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:01.244297 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3470621-0c4e-46fd-ae8c-d88b60a55915 service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:01.361746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:27:01.384932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:01.386202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:01.386700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:01.387201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] PATH_INFO: `/v3/projects/65cb45d314f743799145a0fc614ce83e` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:01.387549 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d3a5ed-85fe-4c3f-b817-32feae387adc admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:01.397894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65cb45d314f743799145a0fc614ce83e)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:01.397894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:01.398234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb913f8-8b73-4a68-a379-b0f4773e2ecd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:01.411779 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb913f8-8b73-4a68-a379-b0f4773e2ecd service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:01.413018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb913f8-8b73-4a68-a379-b0f4773e2ecd service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:01.413454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb913f8-8b73-4a68-a379-b0f4773e2ecd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:01.426517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5bb913f8-8b73-4a68-a379-b0f4773e2ecd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:01.440226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb913f8-8b73-4a68-a379-b0f4773e2ecd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:01.444418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb913f8-8b73-4a68-a379-b0f4773e2ecd service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:01.478244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1006/2017] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:27:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:27:01.542650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65cb45d314f743799145a0fc614ce83e', 'request_id': 'req-1d7babc4-41ed-4c48-8430-366ea43da6e8'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:01.673368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65cb45d314f743799145a0fc614ce83e', 'request_id': 'req-1d7babc4-41ed-4c48-8430-366ea43da6e8'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:01.681587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65cb45d314f743799145a0fc614ce83e', 'request_id': 'req-1d7babc4-41ed-4c48-8430-366ea43da6e8'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:01.699803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] The token cache is being invalidate because project 65cb45d314f743799145a0fc614ce83e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:27:01.700356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:01.711557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d7babc4-41ed-4c48-8430-366ea43da6e8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:27:01.721520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1012/2018] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:27:01 2022] DELETE /identity/v3/projects/65cb45d314f743799145a0fc614ce83e => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:27:01.723361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c41db9a-8cfb-42e1-b841-56194e12be2f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:01.724760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c41db9a-8cfb-42e1-b841-56194e12be2f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:01.725243 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c41db9a-8cfb-42e1-b841-56194e12be2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:01.749507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-1c41db9a-8cfb-42e1-b841-56194e12be2f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:27:01.751361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1c41db9a-8cfb-42e1-b841-56194e12be2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:01.840649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1007/2019] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:27:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:27:01.842539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa5a4d7-44f6-4ac4-94db-47e5c9a01c04 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:01.844093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa5a4d7-44f6-4ac4-94db-47e5c9a01c04 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:01.844476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa5a4d7-44f6-4ac4-94db-47e5c9a01c04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:01.871120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-daa5a4d7-44f6-4ac4-94db-47e5c9a01c04 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:27:01.876923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-daa5a4d7-44f6-4ac4-94db-47e5c9a01c04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:01.903011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1013/2020] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:27:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:27:01.904950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c41db9a-8cfb-42e1-b841-56194e12be2f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:01.925414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c41db9a-8cfb-42e1-b841-56194e12be2f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:01.930434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:01.931664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:01.931949 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:01.932224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:01.932879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:27:01.933181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:01.934823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:01.954820 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1008/2021] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:27:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:27:01.963635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa5a4d7-44f6-4ac4-94db-47e5c9a01c04 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:01.973920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daa5a4d7-44f6-4ac4-94db-47e5c9a01c04 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:01.979567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7ffb5-58fb-441b-be20-439b8ace7786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:01.980441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7ffb5-58fb-441b-be20-439b8ace7786 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:01.980785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7ffb5-58fb-441b-be20-439b8ace7786 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:01.981125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7ffb5-58fb-441b-be20-439b8ace7786 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:01.981860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a7ffb5-58fb-441b-be20-439b8ace7786 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:01.984008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a7ffb5-58fb-441b-be20-439b8ace7786 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:02.064634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1014/2022] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:27:01 2022] POST /identity/v3/projects => generated 369 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:27:02.068689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:02.079723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cede4a5-cfe9-4edc-a905-b0a7c07c2f2d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:02.151438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d52ab0-d9d4-4f4b-99cd-4be7690e61f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:02.153045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d52ab0-d9d4-4f4b-99cd-4be7690e61f0 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:02.153325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d52ab0-d9d4-4f4b-99cd-4be7690e61f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:02.153603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d52ab0-d9d4-4f4b-99cd-4be7690e61f0 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:02.154241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d52ab0-d9d4-4f4b-99cd-4be7690e61f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:02.158754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d52ab0-d9d4-4f4b-99cd-4be7690e61f0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:02.253843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1009/2023] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:27:02 2022] POST /identity/v3/users => generated 285 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:27:02.258177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7ffb5-58fb-441b-be20-439b8ace7786 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:02.269339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7ffb5-58fb-441b-be20-439b8ace7786 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:02.279337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8c7d69-3cd2-4054-8bb1-1f396c8987fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:02.279337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c7d69-3cd2-4054-8bb1-1f396c8987fa admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:02.279337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c7d69-3cd2-4054-8bb1-1f396c8987fa admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:02.279337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c8c7d69-3cd2-4054-8bb1-1f396c8987fa admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:02.279933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8c7d69-3cd2-4054-8bb1-1f396c8987fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:02.281656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c8c7d69-3cd2-4054-8bb1-1f396c8987fa admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:02.296209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1015/2024] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:27:02 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:27:02.300200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d52ab0-d9d4-4f4b-99cd-4be7690e61f0 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:02.310866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d52ab0-d9d4-4f4b-99cd-4be7690e61f0 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:02.316599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e196903a-244e-497b-a445-0d247d19f31d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:02.317360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e196903a-244e-497b-a445-0d247d19f31d admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:02.317743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e196903a-244e-497b-a445-0d247d19f31d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:02.318118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e196903a-244e-497b-a445-0d247d19f31d admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:02.318881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e196903a-244e-497b-a445-0d247d19f31d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:02.320641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e196903a-244e-497b-a445-0d247d19f31d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:02.325931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=7,keystone:SELECT=92,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:27:02.336673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1010/2025] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:27:02 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:27:02.341291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8c7d69-3cd2-4054-8bb1-1f396c8987fa admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:02.352158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c8c7d69-3cd2-4054-8bb1-1f396c8987fa admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:02.357990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebeee80-0d7e-45fb-923a-695425b83f0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:02.358750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebeee80-0d7e-45fb-923a-695425b83f0e admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:02.359267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebeee80-0d7e-45fb-923a-695425b83f0e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:02.359768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebeee80-0d7e-45fb-923a-695425b83f0e admin admin] PATH_INFO: `/v3/projects/010830aaaaf643c6a5b33d3b55e00eb8/users/6a761477eada4897ba44fa4fb3145a6e/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:02.377653 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebeee80-0d7e-45fb-923a-695425b83f0e admin admin] RBAC: Authorizing `identity:create_grant(project_id=010830aaaaf643c6a5b33d3b55e00eb8, user_id=6a761477eada4897ba44fa4fb3145a6e, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:02.381299 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebeee80-0d7e-45fb-923a-695425b83f0e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:02.489334 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1016/2026] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:27:02 2022] PUT /identity/v3/projects/010830aaaaf643c6a5b33d3b55e00eb8/users/6a761477eada4897ba44fa4fb3145a6e/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:27:02.501616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec8e85-35d8-4863-aac9-c139a52656a5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:02.502144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec8e85-35d8-4863-aac9-c139a52656a5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:02.502383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ec8e85-35d8-4863-aac9-c139a52656a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:02.537147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-20ec8e85-35d8-4863-aac9-c139a52656a5 None None] MFA Rules not processed for user `6a761477eada4897ba44fa4fb3145a6e`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:27:02.539287 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-20ec8e85-35d8-4863-aac9-c139a52656a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:02.626834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1011/2027] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:27:02 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:27:02.630933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebeee80-0d7e-45fb-923a-695425b83f0e admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:02.723741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebeee80-0d7e-45fb-923a-695425b83f0e admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:02.729789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd44002-0a99-41e3-bb47-4216d9ee660a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:02.732857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd44002-0a99-41e3-bb47-4216d9ee660a admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:02.732857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd44002-0a99-41e3-bb47-4216d9ee660a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:02.732857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd44002-0a99-41e3-bb47-4216d9ee660a admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:02.732857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd44002-0a99-41e3-bb47-4216d9ee660a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:02.736114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd44002-0a99-41e3-bb47-4216d9ee660a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:02.865029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1017/2028] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:27:02 2022] POST /identity/v3/projects => generated 369 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:27:02.868904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec8e85-35d8-4863-aac9-c139a52656a5 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:02.880402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ec8e85-35d8-4863-aac9-c139a52656a5 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:02.952185 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dceea0-3175-4d2f-9b33-cd9b70fd71c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:02.953060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dceea0-3175-4d2f-9b33-cd9b70fd71c1 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:02.953536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dceea0-3175-4d2f-9b33-cd9b70fd71c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:02.954007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dceea0-3175-4d2f-9b33-cd9b70fd71c1 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:02.954649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dceea0-3175-4d2f-9b33-cd9b70fd71c1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:02.956634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5dceea0-3175-4d2f-9b33-cd9b70fd71c1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:03.081480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1012/2029] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:27:02 2022] POST /identity/v3/users => generated 291 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:27:03.085199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd44002-0a99-41e3-bb47-4216d9ee660a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:03.092575 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:27:03.096529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd44002-0a99-41e3-bb47-4216d9ee660a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:03.102329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7658483b-076b-46fb-aa7d-e881c6697c02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:03.103685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7658483b-076b-46fb-aa7d-e881c6697c02 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:03.103685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7658483b-076b-46fb-aa7d-e881c6697c02 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:03.103685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7658483b-076b-46fb-aa7d-e881c6697c02 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:03.104316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7658483b-076b-46fb-aa7d-e881c6697c02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:03.106009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7658483b-076b-46fb-aa7d-e881c6697c02 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:03.120181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1018/2030] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:27:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:27:03.125202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dceea0-3175-4d2f-9b33-cd9b70fd71c1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:03.138371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5dceea0-3175-4d2f-9b33-cd9b70fd71c1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:03.146044 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b44d28-201c-4337-830f-7b8fc0e03bf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:03.157650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b44d28-201c-4337-830f-7b8fc0e03bf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:03.157650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b44d28-201c-4337-830f-7b8fc0e03bf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:03.157650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b44d28-201c-4337-830f-7b8fc0e03bf1 admin admin] PATH_INFO: `/v3/projects/82ad713fa51f4c93a2193cea591c5f05/users/096dbef3137041c2bec0621bf2153a6d/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:03.169102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b44d28-201c-4337-830f-7b8fc0e03bf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82ad713fa51f4c93a2193cea591c5f05, user_id=096dbef3137041c2bec0621bf2153a6d, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:03.172437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b44d28-201c-4337-830f-7b8fc0e03bf1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:03.227901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:27:03.316773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1013/2031] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:27:03 2022] PUT /identity/v3/projects/82ad713fa51f4c93a2193cea591c5f05/users/096dbef3137041c2bec0621bf2153a6d/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:27:03.345916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7658483b-076b-46fb-aa7d-e881c6697c02 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:03.356243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7658483b-076b-46fb-aa7d-e881c6697c02 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:03.438804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217ac4f2-0904-4576-8557-cf628d799363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:03.439583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ac4f2-0904-4576-8557-cf628d799363 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:03.439931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ac4f2-0904-4576-8557-cf628d799363 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:03.440277 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ac4f2-0904-4576-8557-cf628d799363 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:03.441002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217ac4f2-0904-4576-8557-cf628d799363 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:03.442866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217ac4f2-0904-4576-8557-cf628d799363 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:03.454348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1019/2032] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:27:03 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:27:03.458150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b44d28-201c-4337-830f-7b8fc0e03bf1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:03.469580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b44d28-201c-4337-830f-7b8fc0e03bf1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:03.477315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f469014-3bee-4d71-954e-b0fd73a9c5c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:03.482881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f469014-3bee-4d71-954e-b0fd73a9c5c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:03.482881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f469014-3bee-4d71-954e-b0fd73a9c5c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:03.482881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f469014-3bee-4d71-954e-b0fd73a9c5c4 admin admin] PATH_INFO: `/v3/projects/82ad713fa51f4c93a2193cea591c5f05/users/096dbef3137041c2bec0621bf2153a6d/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:03.488176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f469014-3bee-4d71-954e-b0fd73a9c5c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82ad713fa51f4c93a2193cea591c5f05, user_id=096dbef3137041c2bec0621bf2153a6d, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:03.491110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f469014-3bee-4d71-954e-b0fd73a9c5c4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:03.502851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1014/2033] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:27:03 2022] PUT /identity/v3/projects/82ad713fa51f4c93a2193cea591c5f05/users/096dbef3137041c2bec0621bf2153a6d/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:27:03.513386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc2c25-f322-4b51-87d0-3337368578cd None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:03.517517 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc2c25-f322-4b51-87d0-3337368578cd None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:03.517517 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc2c25-f322-4b51-87d0-3337368578cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:03.549587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-fcbc2c25-f322-4b51-87d0-3337368578cd None None] MFA Rules not processed for user `096dbef3137041c2bec0621bf2153a6d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:27:03.551125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fcbc2c25-f322-4b51-87d0-3337368578cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:03.643439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1020/2034] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:27:03 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:27:05.508905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6516b7-7d73-470b-8cbe-1d170e505455 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:05.509433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6516b7-7d73-470b-8cbe-1d170e505455 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:05.509433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6516b7-7d73-470b-8cbe-1d170e505455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:05.547040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-af6516b7-7d73-470b-8cbe-1d170e505455 None None] MFA Rules not processed for user `84c32bcd9bf94ae2b29548d4411884cd`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:27:05.548289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-af6516b7-7d73-470b-8cbe-1d170e505455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:05.624628 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1015/2035] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:27:05 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:27:05.639830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbc2c25-f322-4b51-87d0-3337368578cd None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:05.641918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fcbc2c25-f322-4b51-87d0-3337368578cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:05.762809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbc2c25-f322-4b51-87d0-3337368578cd None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:05.771796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:05.772552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:05.772829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:05.773110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:05.774972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:05.791819 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:27:05.861805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:05.863361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:05.887099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1021/2036] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:27:05 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:27:09.065818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5d1b0-2a7f-4724-88d7-12271b4624bc None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:09.066316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5d1b0-2a7f-4724-88d7-12271b4624bc None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:09.066496 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d5d1b0-2a7f-4724-88d7-12271b4624bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:09.096947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-71d5d1b0-2a7f-4724-88d7-12271b4624bc None None] MFA Rules not processed for user `11c2166928f8414f8ae545f7bbd9fd6f`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:27:09.103561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-71d5d1b0-2a7f-4724-88d7-12271b4624bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:09.110765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1016/2037] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:27:09 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:27:15.870246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=23 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:27:19.083526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=48,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:27:49.614199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:27:49.615655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:49.631183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c55407-205b-4b09-9985-b67a578e8842 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:27:49.637351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31a9366e-182b-4973-98b7-ddab28f4e5ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:27:49.637951 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a9366e-182b-4973-98b7-ddab28f4e5ee service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:27:49.638228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a9366e-182b-4973-98b7-ddab28f4e5ee service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:27:49.638499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31a9366e-182b-4973-98b7-ddab28f4e5ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:27:49.640205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-31a9366e-182b-4973-98b7-ddab28f4e5ee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:27:49.708765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a9366e-182b-4973-98b7-ddab28f4e5ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:27:49.710349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31a9366e-182b-4973-98b7-ddab28f4e5ee service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:27:49.719234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1022/2038] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:27:49 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:27:59.715089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=33 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:03.695936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5d1b0-2a7f-4724-88d7-12271b4624bc None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:03.758365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d5d1b0-2a7f-4724-88d7-12271b4624bc None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:03.762758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:03.764021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:03.764315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:03.764587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] PATH_INFO: `/v3/users/6a761477eada4897ba44fa4fb3145a6e` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:03.765684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a761477eada4897ba44fa4fb3145a6e)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:03.767622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:03.982191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a761477eada4897ba44fa4fb3145a6e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:03.982768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a761477eada4897ba44fa4fb3145a6e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:03.994689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a761477eada4897ba44fa4fb3145a6e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:04.002099 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a761477eada4897ba44fa4fb3145a6e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:04.002725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a761477eada4897ba44fa4fb3145a6e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:04.071896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1017/2039] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:28:03 2022] DELETE /identity/v3/users/6a761477eada4897ba44fa4fb3145a6e => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:04.081344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f1986-ecfc-4246-9459-67c65b0c8481 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:04.081808 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f1986-ecfc-4246-9459-67c65b0c8481 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:04.081903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3f1986-ecfc-4246-9459-67c65b0c8481 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:04.101533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1f3f1986-ecfc-4246-9459-67c65b0c8481 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:04.102735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1f3f1986-ecfc-4246-9459-67c65b0c8481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:04.159881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1023/2040] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:04.169676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:04.171125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:04.242337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97352bcd-98f7-41cc-a8fb-2b6a4a8c2458 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:04.245940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb001c5-206a-4827-a0ff-8bede09accec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:04.246546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb001c5-206a-4827-a0ff-8bede09accec service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:04.246825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb001c5-206a-4827-a0ff-8bede09accec service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:04.247135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb001c5-206a-4827-a0ff-8bede09accec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:04.256124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb001c5-206a-4827-a0ff-8bede09accec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:04.257551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfb001c5-206a-4827-a0ff-8bede09accec service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:04.267597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1018/2041] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:04.616611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f1986-ecfc-4246-9459-67c65b0c8481 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:04.618192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1f3f1986-ecfc-4246-9459-67c65b0c8481 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:04.630778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3f1986-ecfc-4246-9459-67c65b0c8481 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:04.635476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:04.636232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:04.636524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:04.636800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] PATH_INFO: `/v3/projects/010830aaaaf643c6a5b33d3b55e00eb8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:04.638618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=010830aaaaf643c6a5b33d3b55e00eb8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:04.640582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:04.719024 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '010830aaaaf643c6a5b33d3b55e00eb8', 'request_id': 'req-85589dd9-ecad-429d-82ea-b9574508cee2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:04.797953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '010830aaaaf643c6a5b33d3b55e00eb8', 'request_id': 'req-85589dd9-ecad-429d-82ea-b9574508cee2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:04.803986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '010830aaaaf643c6a5b33d3b55e00eb8', 'request_id': 'req-85589dd9-ecad-429d-82ea-b9574508cee2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:04.815894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] The token cache is being invalidate because project 010830aaaaf643c6a5b33d3b55e00eb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:28:04.816187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:04.816826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:04.818344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1024/2042] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:28:04 2022] DELETE /identity/v3/projects/010830aaaaf643c6a5b33d3b55e00eb8 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:04.822479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb001c5-206a-4827-a0ff-8bede09accec service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:04.832343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfb001c5-206a-4827-a0ff-8bede09accec service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:04.840068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:04.888223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:04.888843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:04.889122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:04.889397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] PATH_INFO: `/v3/users/096dbef3137041c2bec0621bf2153a6d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:04.890504 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=096dbef3137041c2bec0621bf2153a6d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:04.892489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:05.034231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '096dbef3137041c2bec0621bf2153a6d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.034889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '096dbef3137041c2bec0621bf2153a6d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.046883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '096dbef3137041c2bec0621bf2153a6d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.053595 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '096dbef3137041c2bec0621bf2153a6d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.054230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-bc173716-3df3-49e1-ab5c-23f233668f04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '096dbef3137041c2bec0621bf2153a6d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.140487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1019/2043] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:28:04 2022] DELETE /identity/v3/users/096dbef3137041c2bec0621bf2153a6d => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:05.520301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:05.521793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:05.534920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85589dd9-ecad-429d-82ea-b9574508cee2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:05.591880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:05.592444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:05.592721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:05.592991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] PATH_INFO: `/v3/projects/82ad713fa51f4c93a2193cea591c5f05` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:05.594113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82ad713fa51f4c93a2193cea591c5f05)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:05.596064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:05.667704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:05.684107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82ad713fa51f4c93a2193cea591c5f05', 'request_id': 'req-f30c442e-8676-4107-81fc-bf5e849ed751'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.763600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82ad713fa51f4c93a2193cea591c5f05', 'request_id': 'req-f30c442e-8676-4107-81fc-bf5e849ed751'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.769130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82ad713fa51f4c93a2193cea591c5f05', 'request_id': 'req-f30c442e-8676-4107-81fc-bf5e849ed751'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.780732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] The token cache is being invalidate because project 82ad713fa51f4c93a2193cea591c5f05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:28:05.784419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.784419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f30c442e-8676-4107-81fc-bf5e849ed751 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:05.784419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1025/2044] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:28:05 2022] DELETE /identity/v3/projects/82ad713fa51f4c93a2193cea591c5f05 => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:28:05.797036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ae49ce-39c0-4b5b-9d17-42158fde8b92 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:05.797550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ae49ce-39c0-4b5b-9d17-42158fde8b92 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:05.798038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ae49ce-39c0-4b5b-9d17-42158fde8b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:05.819830 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e9ae49ce-39c0-4b5b-9d17-42158fde8b92 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:05.821020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e9ae49ce-39c0-4b5b-9d17-42158fde8b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:05.879676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1020/2045] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:05.887456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2de9cb-4aa5-4fc0-8d40-941f0a056ecf None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:05.887737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2de9cb-4aa5-4fc0-8d40-941f0a056ecf None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:05.888543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2de9cb-4aa5-4fc0-8d40-941f0a056ecf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:05.908347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-9f2de9cb-4aa5-4fc0-8d40-941f0a056ecf None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:05.909528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9f2de9cb-4aa5-4fc0-8d40-941f0a056ecf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:05.917291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1026/2046] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:05.921455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ae49ce-39c0-4b5b-9d17-42158fde8b92 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:05.931143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ae49ce-39c0-4b5b-9d17-42158fde8b92 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:05.936271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:05.936869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:05.937147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:05.937415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:05.938059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:28:05.938358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:05.939998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:05.952282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1021/2047] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:28:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:28:05.956345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2de9cb-4aa5-4fc0-8d40-941f0a056ecf None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:05.965485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2de9cb-4aa5-4fc0-8d40-941f0a056ecf None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:05.970040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bce960-ee91-4a09-812f-eb1cb1c26301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:05.970622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bce960-ee91-4a09-812f-eb1cb1c26301 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:05.970893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bce960-ee91-4a09-812f-eb1cb1c26301 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:05.971202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bce960-ee91-4a09-812f-eb1cb1c26301 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:05.971821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bce960-ee91-4a09-812f-eb1cb1c26301 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:05.973758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bce960-ee91-4a09-812f-eb1cb1c26301 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:06.084938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1027/2048] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:28:05 2022] POST /identity/v3/projects => generated 371 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:28:06.089002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:06.098973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3610a040-7d22-483d-b8ca-47d3b614c247 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:06.111154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:06.154673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8eacd0-b7d1-4c8d-8cb2-e7275c3dc095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:06.155235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8eacd0-b7d1-4c8d-8cb2-e7275c3dc095 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:06.155514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8eacd0-b7d1-4c8d-8cb2-e7275c3dc095 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:06.155785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8eacd0-b7d1-4c8d-8cb2-e7275c3dc095 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:06.156404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8eacd0-b7d1-4c8d-8cb2-e7275c3dc095 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:06.158415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8eacd0-b7d1-4c8d-8cb2-e7275c3dc095 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:06.238096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1022/2049] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:28:06 2022] POST /identity/v3/users => generated 286 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:28:06.242336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bce960-ee91-4a09-812f-eb1cb1c26301 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:06.253563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bce960-ee91-4a09-812f-eb1cb1c26301 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:06.258226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bd6b92-99db-461d-b63f-1a029f539a66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:06.258813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bd6b92-99db-461d-b63f-1a029f539a66 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:06.259124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bd6b92-99db-461d-b63f-1a029f539a66 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:06.259418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6bd6b92-99db-461d-b63f-1a029f539a66 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:06.260062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bd6b92-99db-461d-b63f-1a029f539a66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:06.261648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6bd6b92-99db-461d-b63f-1a029f539a66 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:06.272015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1028/2050] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:28:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:28:06.275874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8eacd0-b7d1-4c8d-8cb2-e7275c3dc095 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:06.285764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8eacd0-b7d1-4c8d-8cb2-e7275c3dc095 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:06.290284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b23b78-5dcc-412b-a743-bc4e42483ba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:06.290873 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b23b78-5dcc-412b-a743-bc4e42483ba8 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:06.291186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b23b78-5dcc-412b-a743-bc4e42483ba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:06.291458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b23b78-5dcc-412b-a743-bc4e42483ba8 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:06.292090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b23b78-5dcc-412b-a743-bc4e42483ba8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:06.293668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46b23b78-5dcc-412b-a743-bc4e42483ba8 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:06.304045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1023/2051] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:28:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:28:06.308005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bd6b92-99db-461d-b63f-1a029f539a66 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:06.318531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6bd6b92-99db-461d-b63f-1a029f539a66 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:06.323202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:06.323843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:06.324124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:06.324428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] PATH_INFO: `/v3/projects/a9b54df9dfd24a61832dcbc735594aa5/users/8b08a42c389049478ae7aec78d9752cc/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:06.339597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9b54df9dfd24a61832dcbc735594aa5, user_id=8b08a42c389049478ae7aec78d9752cc, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:06.342259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:06.392208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1029/2052] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:28:06 2022] PUT /identity/v3/projects/a9b54df9dfd24a61832dcbc735594aa5/users/8b08a42c389049478ae7aec78d9752cc/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:28:06.403486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f97508-bd1f-476a-a684-f521e27858fb None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:06.407668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f97508-bd1f-476a-a684-f521e27858fb None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:06.407668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f97508-bd1f-476a-a684-f521e27858fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:06.431537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-f4f97508-bd1f-476a-a684-f521e27858fb None None] MFA Rules not processed for user `8b08a42c389049478ae7aec78d9752cc`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:06.432730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f4f97508-bd1f-476a-a684-f521e27858fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:06.495423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1024/2053] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:06 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:16.387976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=6,keystone:SELECT=41,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:16.492539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=58,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:16.627672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:16.629466 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:16.693781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4249c-24da-452b-9e14-92d2306923ce admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:16.698433 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:16.699091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:16.699378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:16.699657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] PATH_INFO: `/v3/users/11c2166928f8414f8ae545f7bbd9fd6f` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:16.700977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=11c2166928f8414f8ae545f7bbd9fd6f)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:16.704012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:16.897827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11c2166928f8414f8ae545f7bbd9fd6f'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:16.980348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11c2166928f8414f8ae545f7bbd9fd6f'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:16.993346 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11c2166928f8414f8ae545f7bbd9fd6f'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.000280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11c2166928f8414f8ae545f7bbd9fd6f'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.000930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11c2166928f8414f8ae545f7bbd9fd6f'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.006775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1030/2054] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:28:16 2022] DELETE /identity/v3/users/11c2166928f8414f8ae545f7bbd9fd6f => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:17.222260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f97508-bd1f-476a-a684-f521e27858fb None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:17.234021 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f97508-bd1f-476a-a684-f521e27858fb None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:17.290080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:17.290708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:17.291018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:17.291299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] PATH_INFO: `/v3/projects/0b5ba219e7b44b228ed013f830a61572` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:17.292420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0b5ba219e7b44b228ed013f830a61572)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:17.294326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:17.416391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0b5ba219e7b44b228ed013f830a61572', 'request_id': 'req-eccd19ad-4a82-49a2-a652-0a64208eee55'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.422595 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0b5ba219e7b44b228ed013f830a61572', 'request_id': 'req-eccd19ad-4a82-49a2-a652-0a64208eee55'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.428108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0b5ba219e7b44b228ed013f830a61572', 'request_id': 'req-eccd19ad-4a82-49a2-a652-0a64208eee55'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.511889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] The token cache is being invalidate because project 0b5ba219e7b44b228ed013f830a61572 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:28:17.512406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.512788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.515077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1025/2055] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:28:17 2022] DELETE /identity/v3/projects/0b5ba219e7b44b228ed013f830a61572 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:17.519185 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:17.522017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:17.533521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad70e5bc-ef1a-481c-867a-0804f6e581a2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:17.588549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:17.589164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:17.589436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:17.589705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] PATH_INFO: `/v3/users/84c32bcd9bf94ae2b29548d4411884cd` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:17.590810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=84c32bcd9bf94ae2b29548d4411884cd)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:17.592877 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:17.613450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:17.795199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84c32bcd9bf94ae2b29548d4411884cd'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.866531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84c32bcd9bf94ae2b29548d4411884cd'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.879392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84c32bcd9bf94ae2b29548d4411884cd'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.886489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84c32bcd9bf94ae2b29548d4411884cd'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.887172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-fc27313a-1971-41be-83ee-a2af61589dd0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84c32bcd9bf94ae2b29548d4411884cd'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:17.893501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1031/2056] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:28:17 2022] DELETE /identity/v3/users/84c32bcd9bf94ae2b29548d4411884cd => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:18.154272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:18.155711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:18.171283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eccd19ad-4a82-49a2-a652-0a64208eee55 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:18.228323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:18.228944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.229222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.229491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] PATH_INFO: `/v3/projects/2c3ddfd7078c4621b2c42181adcbaf19` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:18.230585 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2c3ddfd7078c4621b2c42181adcbaf19)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:18.233420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:18.347468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:18.360163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2c3ddfd7078c4621b2c42181adcbaf19', 'request_id': 'req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:18.366301 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2c3ddfd7078c4621b2c42181adcbaf19', 'request_id': 'req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:18.371687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2c3ddfd7078c4621b2c42181adcbaf19', 'request_id': 'req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:18.448823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] The token cache is being invalidate because project 2c3ddfd7078c4621b2c42181adcbaf19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:28:18.449195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:18.453480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-8cc61103-2faf-4543-8d05-e47f1b8e3a3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:18.453480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1026/2057] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:28:18 2022] DELETE /identity/v3/projects/2c3ddfd7078c4621b2c42181adcbaf19 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:18.457468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e189b217-5289-4a84-88d8-01ab5fdb46fc None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.457778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e189b217-5289-4a84-88d8-01ab5fdb46fc None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.458053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e189b217-5289-4a84-88d8-01ab5fdb46fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:18.478168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e189b217-5289-4a84-88d8-01ab5fdb46fc None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:18.479398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e189b217-5289-4a84-88d8-01ab5fdb46fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:18.538113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1032/2058] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:18.547232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153188e7-f7ca-486e-8127-ae676ef5666c None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.547689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153188e7-f7ca-486e-8127-ae676ef5666c None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.547835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153188e7-f7ca-486e-8127-ae676ef5666c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:18.571351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-153188e7-f7ca-486e-8127-ae676ef5666c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:18.572600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-153188e7-f7ca-486e-8127-ae676ef5666c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:18.581608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1027/2059] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:18.585806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e189b217-5289-4a84-88d8-01ab5fdb46fc None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:18.596211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e189b217-5289-4a84-88d8-01ab5fdb46fc None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:18.601036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:18.601660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.601963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.602256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:18.602907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:28:18.603235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:18.604858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:18.617053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1033/2060] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:28:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:28:18.621248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153188e7-f7ca-486e-8127-ae676ef5666c None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:18.630415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153188e7-f7ca-486e-8127-ae676ef5666c None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:18.635073 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f69046e-4078-4977-b9c5-8916b8903e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:18.635670 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69046e-4078-4977-b9c5-8916b8903e0b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.635951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69046e-4078-4977-b9c5-8916b8903e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.636224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f69046e-4078-4977-b9c5-8916b8903e0b admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:18.636856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f69046e-4078-4977-b9c5-8916b8903e0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:18.638747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f69046e-4078-4977-b9c5-8916b8903e0b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:18.725737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1028/2061] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:28:18 2022] POST /identity/v3/projects => generated 377 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:28:18.729577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:18.741863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6828df2-33cf-406c-9f07-19c2322b7f0e admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:18.799990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98949007-1c8b-4a77-8fab-49fb1c5e3a94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:18.800597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98949007-1c8b-4a77-8fab-49fb1c5e3a94 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.800874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98949007-1c8b-4a77-8fab-49fb1c5e3a94 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.801147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98949007-1c8b-4a77-8fab-49fb1c5e3a94 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:18.801823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98949007-1c8b-4a77-8fab-49fb1c5e3a94 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:18.803816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98949007-1c8b-4a77-8fab-49fb1c5e3a94 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:18.901125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1034/2062] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:28:18 2022] POST /identity/v3/users => generated 289 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:28:18.905931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f69046e-4078-4977-b9c5-8916b8903e0b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:18.916410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f69046e-4078-4977-b9c5-8916b8903e0b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:18.921070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f96ab1-fcff-427e-b7bd-a04648dcd22c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:18.921679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f96ab1-fcff-427e-b7bd-a04648dcd22c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.921960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f96ab1-fcff-427e-b7bd-a04648dcd22c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.922233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f96ab1-fcff-427e-b7bd-a04648dcd22c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:18.922905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f96ab1-fcff-427e-b7bd-a04648dcd22c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:18.924542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f96ab1-fcff-427e-b7bd-a04648dcd22c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:18.935938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1029/2063] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:28:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:28:18.939829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98949007-1c8b-4a77-8fab-49fb1c5e3a94 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:18.950252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98949007-1c8b-4a77-8fab-49fb1c5e3a94 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:18.955748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249473aa-5b65-406c-99fe-1cef1d528d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:18.956338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249473aa-5b65-406c-99fe-1cef1d528d67 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.956612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249473aa-5b65-406c-99fe-1cef1d528d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.956881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249473aa-5b65-406c-99fe-1cef1d528d67 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:18.957526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249473aa-5b65-406c-99fe-1cef1d528d67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:18.959169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249473aa-5b65-406c-99fe-1cef1d528d67 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:18.972842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1035/2064] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:28:18 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:28:18.974227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f96ab1-fcff-427e-b7bd-a04648dcd22c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:18.985081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f96ab1-fcff-427e-b7bd-a04648dcd22c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:18.989910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:18.990563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:18.990841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:18.991245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] PATH_INFO: `/v3/projects/9b8d72b42f384b4a9933c4648913857c/users/690632c6d7494c22997ac41257ad32ff/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:19.006200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=9b8d72b42f384b4a9933c4648913857c, user_id=690632c6d7494c22997ac41257ad32ff, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:19.008875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:19.101004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1030/2065] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:28:18 2022] PUT /identity/v3/projects/9b8d72b42f384b4a9933c4648913857c/users/690632c6d7494c22997ac41257ad32ff/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:19.106045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8745f0ea-c5e8-4c9e-8c23-b4631a4d2ace None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:19.106332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8745f0ea-c5e8-4c9e-8c23-b4631a4d2ace None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:19.106603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8745f0ea-c5e8-4c9e-8c23-b4631a4d2ace None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:19.114561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:19.125879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8745f0ea-c5e8-4c9e-8c23-b4631a4d2ace None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:19.127117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8745f0ea-c5e8-4c9e-8c23-b4631a4d2ace None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:19.184845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1036/2066] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:19.195198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:19.196933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:19.254022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3710c7-e2fc-4e74-acbf-97624b4f2d4b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:19.257555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd7e7bc-af3f-4191-99d1-048d920ceac8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:19.258150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd7e7bc-af3f-4191-99d1-048d920ceac8 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:19.258429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd7e7bc-af3f-4191-99d1-048d920ceac8 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:19.258699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd7e7bc-af3f-4191-99d1-048d920ceac8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:19.268683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd7e7bc-af3f-4191-99d1-048d920ceac8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:19.270119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd7e7bc-af3f-4191-99d1-048d920ceac8 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:19.279784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1031/2067] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:24.286227 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae868987-969a-4952-ae94-ad24b71a6cca None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:24.286691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae868987-969a-4952-ae94-ad24b71a6cca None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:24.286781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae868987-969a-4952-ae94-ad24b71a6cca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:24.319896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ae868987-969a-4952-ae94-ad24b71a6cca None None] MFA Rules not processed for user `690632c6d7494c22997ac41257ad32ff`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:24.321417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ae868987-969a-4952-ae94-ad24b71a6cca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:24.403806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1037/2068] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:24.408580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd7e7bc-af3f-4191-99d1-048d920ceac8 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:24.419512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd7e7bc-af3f-4191-99d1-048d920ceac8 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:24.424909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917e90f3-6a51-4895-bd2e-5899cc98fd3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:24.425508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917e90f3-6a51-4895-bd2e-5899cc98fd3f admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:24.425966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917e90f3-6a51-4895-bd2e-5899cc98fd3f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:24.426263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917e90f3-6a51-4895-bd2e-5899cc98fd3f admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:24.426906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917e90f3-6a51-4895-bd2e-5899cc98fd3f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:24.429015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917e90f3-6a51-4895-bd2e-5899cc98fd3f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:24.520798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1032/2069] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:28:24 2022] POST /identity/v3/projects => generated 379 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:28:24.524828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae868987-969a-4952-ae94-ad24b71a6cca None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:24.537681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae868987-969a-4952-ae94-ad24b71a6cca None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:24.601584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106f404e-7675-4f78-9e85-88375df100b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:24.602236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106f404e-7675-4f78-9e85-88375df100b5 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:24.602476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106f404e-7675-4f78-9e85-88375df100b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:24.602749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106f404e-7675-4f78-9e85-88375df100b5 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:24.603595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106f404e-7675-4f78-9e85-88375df100b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:24.605601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106f404e-7675-4f78-9e85-88375df100b5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:24.709645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1038/2070] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:28:24 2022] POST /identity/v3/users => generated 296 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:28:24.709983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917e90f3-6a51-4895-bd2e-5899cc98fd3f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:24.723880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917e90f3-6a51-4895-bd2e-5899cc98fd3f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:24.729251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d1fdfc-20db-485a-a141-7cdf0fa997a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:24.729674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d1fdfc-20db-485a-a141-7cdf0fa997a4 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:24.729991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d1fdfc-20db-485a-a141-7cdf0fa997a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:24.730271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d1fdfc-20db-485a-a141-7cdf0fa997a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:24.730926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d1fdfc-20db-485a-a141-7cdf0fa997a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:24.733160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d1fdfc-20db-485a-a141-7cdf0fa997a4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:24.745408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1033/2071] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:28:24 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:28:24.749733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106f404e-7675-4f78-9e85-88375df100b5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:24.760399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106f404e-7675-4f78-9e85-88375df100b5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:24.767360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39d25a4-48b8-4fab-9761-da7789e687f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:24.768031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d25a4-48b8-4fab-9761-da7789e687f2 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:24.768303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d25a4-48b8-4fab-9761-da7789e687f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:24.768570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d25a4-48b8-4fab-9761-da7789e687f2 admin admin] PATH_INFO: `/v3/projects/787f90130cd842138b0c221e92d82f65/users/812c54da9da240678f255ee5e192f5eb/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:24.787269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39d25a4-48b8-4fab-9761-da7789e687f2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=787f90130cd842138b0c221e92d82f65, user_id=812c54da9da240678f255ee5e192f5eb, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:24.789963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39d25a4-48b8-4fab-9761-da7789e687f2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:24.850699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:24.855601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1039/2072] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:28:24 2022] PUT /identity/v3/projects/787f90130cd842138b0c221e92d82f65/users/812c54da9da240678f255ee5e192f5eb/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:24.859792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d1fdfc-20db-485a-a141-7cdf0fa997a4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:24.869863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d1fdfc-20db-485a-a141-7cdf0fa997a4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:24.893408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:24.925008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b278ac8-3620-4cae-9d49-73f7e36a585f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:24.925749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b278ac8-3620-4cae-9d49-73f7e36a585f admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:24.925882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b278ac8-3620-4cae-9d49-73f7e36a585f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:24.926154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b278ac8-3620-4cae-9d49-73f7e36a585f admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:24.926799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b278ac8-3620-4cae-9d49-73f7e36a585f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:24.928438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b278ac8-3620-4cae-9d49-73f7e36a585f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:24.939259 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1034/2073] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:28:24 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:28:24.943591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39d25a4-48b8-4fab-9761-da7789e687f2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:24.955191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39d25a4-48b8-4fab-9761-da7789e687f2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:24.959859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d65320-9b9e-4bd3-ba3f-db2bc3ed212e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:24.960485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d65320-9b9e-4bd3-ba3f-db2bc3ed212e admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:24.960758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d65320-9b9e-4bd3-ba3f-db2bc3ed212e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:24.961054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d65320-9b9e-4bd3-ba3f-db2bc3ed212e admin admin] PATH_INFO: `/v3/projects/787f90130cd842138b0c221e92d82f65/users/812c54da9da240678f255ee5e192f5eb/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:24.963177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d65320-9b9e-4bd3-ba3f-db2bc3ed212e admin admin] RBAC: Authorizing `identity:create_grant(project_id=787f90130cd842138b0c221e92d82f65, user_id=812c54da9da240678f255ee5e192f5eb, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:24.966263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d65320-9b9e-4bd3-ba3f-db2bc3ed212e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:24.977421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1040/2074] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:28:24 2022] PUT /identity/v3/projects/787f90130cd842138b0c221e92d82f65/users/812c54da9da240678f255ee5e192f5eb/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:28.708737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b278ac8-3620-4cae-9d49-73f7e36a585f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:28.772856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b278ac8-3620-4cae-9d49-73f7e36a585f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:28.849915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:28.850842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:28.850842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:28.851038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:28.853179 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:28.864633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:28.866163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:28.876478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1035/2075] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:30.696017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b073920-3d89-4861-8633-3717edbeb5e6 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:30.696562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b073920-3d89-4861-8633-3717edbeb5e6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:30.696562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b073920-3d89-4861-8633-3717edbeb5e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:30.726062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3b073920-3d89-4861-8633-3717edbeb5e6 None None] MFA Rules not processed for user `812c54da9da240678f255ee5e192f5eb`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:30.727367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3b073920-3d89-4861-8633-3717edbeb5e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:30.805145 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1041/2076] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:30 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:30.826032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:30.828253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:30.909169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dcc210-c064-49cb-9a54-7958e5bdf3d9 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:30.914778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd46f686-5386-441d-a2dd-5098d288cce8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:30.915439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd46f686-5386-441d-a2dd-5098d288cce8 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:30.915721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd46f686-5386-441d-a2dd-5098d288cce8 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:30.915995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd46f686-5386-441d-a2dd-5098d288cce8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:30.933475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:30.979068 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd46f686-5386-441d-a2dd-5098d288cce8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:30.980680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd46f686-5386-441d-a2dd-5098d288cce8 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:30.990926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1036/2077] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:30 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:40.802022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=39,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:40.986335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=19 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:42.162927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b073920-3d89-4861-8633-3717edbeb5e6 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:42.164337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3b073920-3d89-4861-8633-3717edbeb5e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:42.263824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b073920-3d89-4861-8633-3717edbeb5e6 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:42.268222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:42.269489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:42.269773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:42.270045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:42.271825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:42.365501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:42.366939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:42.385316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1042/2078] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 223 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:42.427243 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd46f686-5386-441d-a2dd-5098d288cce8 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:42.449695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd46f686-5386-441d-a2dd-5098d288cce8 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:42.452363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:42.453207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:42.453487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:42.453734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:42.455441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] Could not find project: 82ad713fa51f4c93a2193cea591c5f05.: keystone.exception.ProjectNotFound: Could not find project: 82ad713fa51f4c93a2193cea591c5f05. Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Oct 19 22:28:42.492131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 22:28:42.493045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self._validate_token_resources() Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 22:28:42.493967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application with Lock( Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._enter() Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.creator() Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application created_value = creator( Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 22:28:42.494875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 22:28:42.495941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: 82ad713fa51f4c93a2193cea591c5f05. Oct 19 22:28:42.495941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: ERROR keystone.server.flask.application Oct 19 22:28:42.495941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1037/2079] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:42 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 67 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Oct 19 22:28:43.066684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:43.075250 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd3be8f-b439-4cb0-9c83-2546d21666e9 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:43.079085 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:43.079676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:43.080388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:43.080388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:43.082484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:43.115924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:43.158401 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:43.159962 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:43.170073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1043/2080] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:43 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:43.399877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:43.401322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:43.413602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7acd342-f413-4973-90a9-ae35cef5593a service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:43.418313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:43.419462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:43.419818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:43.420154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:43.422050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:43.436419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:43.437944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:43.449734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1038/2081] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:43 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:43.663907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:43.678422 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bdc247-7bae-4095-a367-4f54230be5d1 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:43.682966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e61f2f6-e8c1-4a97-bec4-d524b1b3458e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:43.683581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61f2f6-e8c1-4a97-bec4-d524b1b3458e service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:43.683857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61f2f6-e8c1-4a97-bec4-d524b1b3458e service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:43.684130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e61f2f6-e8c1-4a97-bec4-d524b1b3458e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:43.689167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2e61f2f6-e8c1-4a97-bec4-d524b1b3458e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:43.794392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e61f2f6-e8c1-4a97-bec4-d524b1b3458e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:43.797342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e61f2f6-e8c1-4a97-bec4-d524b1b3458e service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:43.812855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1044/2082] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:43 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:43.837773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:43.839666 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:43.946089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7d2b2ac-0676-4a25-8493-ba6f3620e417 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:43.956892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:43.958349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:43.959269 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:43.959734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] PATH_INFO: `/v3/users/8b08a42c389049478ae7aec78d9752cc` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:43.961425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b08a42c389049478ae7aec78d9752cc)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:43.963706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:44.194238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b08a42c389049478ae7aec78d9752cc'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:44.195035 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b08a42c389049478ae7aec78d9752cc'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:44.211202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b08a42c389049478ae7aec78d9752cc'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:44.225059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b08a42c389049478ae7aec78d9752cc'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:44.225059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b08a42c389049478ae7aec78d9752cc'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:44.309638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1039/2083] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:28:43 2022] DELETE /identity/v3/users/8b08a42c389049478ae7aec78d9752cc => generated 0 bytes in 472 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:44.314387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f3914-5dbf-4d2b-b6bc-d89df4e32768 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:44.315428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f3914-5dbf-4d2b-b6bc-d89df4e32768 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:44.316451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10f3914-5dbf-4d2b-b6bc-d89df4e32768 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:44.340773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b10f3914-5dbf-4d2b-b6bc-d89df4e32768 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:44.342147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b10f3914-5dbf-4d2b-b6bc-d89df4e32768 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:44.422997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1045/2084] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:44.434219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:44.435872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:44.513119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:44.517702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0f0eff-57de-41c2-82ff-0b9ef0174b3a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:44.522131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:44.522875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:44.523054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:44.523744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:44.538939 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:44.540458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:44.551909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1040/2085] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:44.686677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10f3914-5dbf-4d2b-b6bc-d89df4e32768 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:44.708144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10f3914-5dbf-4d2b-b6bc-d89df4e32768 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:44.713217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a5de32-45ab-4ea8-93de-142df89cebf9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:44.713667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a5de32-45ab-4ea8-93de-142df89cebf9 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:44.714168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a5de32-45ab-4ea8-93de-142df89cebf9 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:44.714557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a5de32-45ab-4ea8-93de-142df89cebf9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:44.720109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-08a5de32-45ab-4ea8-93de-142df89cebf9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:44.785851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:44.806279 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a5de32-45ab-4ea8-93de-142df89cebf9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:44.807917 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a5de32-45ab-4ea8-93de-142df89cebf9 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:44.821001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1046/2086] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:45.377232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:45.378934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:45.403893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adcac682-0b17-4077-a4d8-1338b39726a6 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:45.410406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:45.411222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:45.411569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:45.411911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] PATH_INFO: `/v3/projects/a9b54df9dfd24a61832dcbc735594aa5` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:45.413315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9b54df9dfd24a61832dcbc735594aa5)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:45.415454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:45.592971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9b54df9dfd24a61832dcbc735594aa5', 'request_id': 'req-ff05368a-e001-4410-8b29-f0bd96c184d8'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:45.605803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9b54df9dfd24a61832dcbc735594aa5', 'request_id': 'req-ff05368a-e001-4410-8b29-f0bd96c184d8'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:45.620790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9b54df9dfd24a61832dcbc735594aa5', 'request_id': 'req-ff05368a-e001-4410-8b29-f0bd96c184d8'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:45.689573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] The token cache is being invalidate because project a9b54df9dfd24a61832dcbc735594aa5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:28:45.690223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:45.691829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff05368a-e001-4410-8b29-f0bd96c184d8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:28:45.693771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1041/2087] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:28:45 2022] DELETE /identity/v3/projects/a9b54df9dfd24a61832dcbc735594aa5 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:45.698712 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d65ec0-e8cc-4ca0-996b-ee930d67aabd None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:45.699259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d65ec0-e8cc-4ca0-996b-ee930d67aabd None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:45.699346 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d65ec0-e8cc-4ca0-996b-ee930d67aabd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:45.721417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-22d65ec0-e8cc-4ca0-996b-ee930d67aabd None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:45.722710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-22d65ec0-e8cc-4ca0-996b-ee930d67aabd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:45.801217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1047/2088] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:28:45.809125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eea3c3-15a2-4874-bb83-3ce06d4202db None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:45.809609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eea3c3-15a2-4874-bb83-3ce06d4202db None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:45.809936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eea3c3-15a2-4874-bb83-3ce06d4202db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:45.839643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a9eea3c3-15a2-4874-bb83-3ce06d4202db None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:45.840887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a9eea3c3-15a2-4874-bb83-3ce06d4202db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:45.855389 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1042/2089] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:45.860148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d65ec0-e8cc-4ca0-996b-ee930d67aabd None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:45.879690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d65ec0-e8cc-4ca0-996b-ee930d67aabd None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:45.903828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:45.904419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:45.904908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:45.905417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:45.906246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:28:45.906691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:45.908517 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:45.932408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eea3c3-15a2-4874-bb83-3ce06d4202db None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:45.940501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1048/2090] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:28:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:28:45.946146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eea3c3-15a2-4874-bb83-3ce06d4202db None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:45.952288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de81cc2-5951-4129-9425-a4412b785d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:45.953260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de81cc2-5951-4129-9425-a4412b785d9e admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:45.953610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de81cc2-5951-4129-9425-a4412b785d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:45.953948 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de81cc2-5951-4129-9425-a4412b785d9e admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:45.954754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de81cc2-5951-4129-9425-a4412b785d9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:45.957612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de81cc2-5951-4129-9425-a4412b785d9e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:46.050532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1043/2091] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:28:45 2022] POST /identity/v3/projects => generated 375 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:28:46.053187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:46.064007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8100a4c-c3f5-4fe3-9f9e-e1ae14969620 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:46.146222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ecc9e9-1cdd-4d81-9cbb-fcc18d4ae357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:46.146222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ecc9e9-1cdd-4d81-9cbb-fcc18d4ae357 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:46.146222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ecc9e9-1cdd-4d81-9cbb-fcc18d4ae357 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:46.146222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ecc9e9-1cdd-4d81-9cbb-fcc18d4ae357 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:46.146544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ecc9e9-1cdd-4d81-9cbb-fcc18d4ae357 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:46.147549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ecc9e9-1cdd-4d81-9cbb-fcc18d4ae357 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:46.228804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de81cc2-5951-4129-9425-a4412b785d9e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:46.236832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1049/2092] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:28:46 2022] POST /identity/v3/users => generated 288 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:28:46.241104 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de81cc2-5951-4129-9425-a4412b785d9e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:46.245980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296465b8-8b41-4760-8e88-e5ec1f35cb97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:46.247468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296465b8-8b41-4760-8e88-e5ec1f35cb97 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:46.247753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296465b8-8b41-4760-8e88-e5ec1f35cb97 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:46.249242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296465b8-8b41-4760-8e88-e5ec1f35cb97 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:46.249242 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296465b8-8b41-4760-8e88-e5ec1f35cb97 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:46.250643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296465b8-8b41-4760-8e88-e5ec1f35cb97 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:46.264176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1044/2093] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:28:46 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:28:46.269692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ecc9e9-1cdd-4d81-9cbb-fcc18d4ae357 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:46.280845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ecc9e9-1cdd-4d81-9cbb-fcc18d4ae357 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:46.286594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df275a4-29e2-495e-ad32-6409c8180cc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:46.287377 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df275a4-29e2-495e-ad32-6409c8180cc5 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:46.287726 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df275a4-29e2-495e-ad32-6409c8180cc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:46.288177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df275a4-29e2-495e-ad32-6409c8180cc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:46.288792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df275a4-29e2-495e-ad32-6409c8180cc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:46.290481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df275a4-29e2-495e-ad32-6409c8180cc5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:46.305900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1050/2094] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:28:46 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:28:46.306876 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296465b8-8b41-4760-8e88-e5ec1f35cb97 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:46.317713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296465b8-8b41-4760-8e88-e5ec1f35cb97 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:46.326510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:46.327262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:46.332317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:46.332317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] PATH_INFO: `/v3/projects/c9922df8fe74462cab223bf2b0d7ce43/users/ca0160bd02954a64ac4dfb3c30697570/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:46.346425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9922df8fe74462cab223bf2b0d7ce43, user_id=ca0160bd02954a64ac4dfb3c30697570, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:46.349737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:46.468369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1045/2095] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:28:46 2022] PUT /identity/v3/projects/c9922df8fe74462cab223bf2b0d7ce43/users/ca0160bd02954a64ac4dfb3c30697570/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:28:46.481189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db53c1-fab9-47e2-bc72-ba8afa780e1d None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:46.481481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db53c1-fab9-47e2-bc72-ba8afa780e1d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:46.481755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db53c1-fab9-47e2-bc72-ba8afa780e1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:46.513868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-d4db53c1-fab9-47e2-bc72-ba8afa780e1d None None] MFA Rules not processed for user `ca0160bd02954a64ac4dfb3c30697570`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:28:46.515314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d4db53c1-fab9-47e2-bc72-ba8afa780e1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:46.572554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:46.601876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1051/2096] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:28:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:28:47.251986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:28:47.254381 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:47.329513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad166116-9cf7-4f6c-913f-f65d15936f6b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:28:47.335092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d33ec7-d23a-4528-b8ef-6abe908dd649 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:28:47.335708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d33ec7-d23a-4528-b8ef-6abe908dd649 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:28:47.336418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d33ec7-d23a-4528-b8ef-6abe908dd649 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:28:47.336418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d33ec7-d23a-4528-b8ef-6abe908dd649 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:28:47.338099 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c6d33ec7-d23a-4528-b8ef-6abe908dd649 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:28:47.341791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:47.414146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d33ec7-d23a-4528-b8ef-6abe908dd649 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:28:47.415793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d33ec7-d23a-4528-b8ef-6abe908dd649 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:28:47.424705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1046/2097] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:28:47 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:28:56.649956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=8 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:28:57.575515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=30 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:29:46.669176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4db53c1-fab9-47e2-bc72-ba8afa780e1d None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:46.730001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4db53c1-fab9-47e2-bc72-ba8afa780e1d None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:46.734478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:46.735119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:46.735397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:46.735667 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] PATH_INFO: `/v3/users/ca0160bd02954a64ac4dfb3c30697570` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:46.736783 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca0160bd02954a64ac4dfb3c30697570)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:46.738695 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:46.920437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca0160bd02954a64ac4dfb3c30697570'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:46.986108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca0160bd02954a64ac4dfb3c30697570'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:46.998183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca0160bd02954a64ac4dfb3c30697570'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:47.005136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca0160bd02954a64ac4dfb3c30697570'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:47.005769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca0160bd02954a64ac4dfb3c30697570'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:47.012175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1052/2098] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:29:46 2022] DELETE /identity/v3/users/ca0160bd02954a64ac4dfb3c30697570 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:29:47.016787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a169d8-26a7-433e-8317-810ee17ec3df None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:47.017271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a169d8-26a7-433e-8317-810ee17ec3df None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:47.017367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a169d8-26a7-433e-8317-810ee17ec3df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:47.037986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-77a169d8-26a7-433e-8317-810ee17ec3df None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:29:47.039272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-77a169d8-26a7-433e-8317-810ee17ec3df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:29:47.101361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1047/2099] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:29:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:29:47.111189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:47.112618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:29:47.169125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec790423-448b-4712-a075-3096492ac942 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:47.172812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b738be-be05-428d-ac47-32247b2dda2f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:47.173343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b738be-be05-428d-ac47-32247b2dda2f service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:47.173616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b738be-be05-428d-ac47-32247b2dda2f service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:47.173881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b738be-be05-428d-ac47-32247b2dda2f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:47.182581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b738be-be05-428d-ac47-32247b2dda2f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:47.184062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09b738be-be05-428d-ac47-32247b2dda2f service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:47.193188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1053/2100] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:29:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:29:47.582435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a169d8-26a7-433e-8317-810ee17ec3df None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:47.584786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-77a169d8-26a7-433e-8317-810ee17ec3df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:29:47.595610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a169d8-26a7-433e-8317-810ee17ec3df None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:47.600349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:47.600953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:47.601250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:47.601526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] PATH_INFO: `/v3/projects/c9922df8fe74462cab223bf2b0d7ce43` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:47.602707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9922df8fe74462cab223bf2b0d7ce43)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:47.605153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:47.718289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9922df8fe74462cab223bf2b0d7ce43', 'request_id': 'req-d339dec5-f512-4197-80e7-003ef6d9d549'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:47.724524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9922df8fe74462cab223bf2b0d7ce43', 'request_id': 'req-d339dec5-f512-4197-80e7-003ef6d9d549'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:47.730721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9922df8fe74462cab223bf2b0d7ce43', 'request_id': 'req-d339dec5-f512-4197-80e7-003ef6d9d549'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:47.797881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] The token cache is being invalidate because project c9922df8fe74462cab223bf2b0d7ce43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:29:47.799463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:47.799463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d339dec5-f512-4197-80e7-003ef6d9d549 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:29:47.800366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1048/2101] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:29:47 2022] DELETE /identity/v3/projects/c9922df8fe74462cab223bf2b0d7ce43 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:29:47.808702 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b8902-d88a-49bb-82a0-0a522b45fe95 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:47.809152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b8902-d88a-49bb-82a0-0a522b45fe95 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:47.809250 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03b8902-d88a-49bb-82a0-0a522b45fe95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:47.828846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b03b8902-d88a-49bb-82a0-0a522b45fe95 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:29:47.830287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b03b8902-d88a-49bb-82a0-0a522b45fe95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:29:47.847373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:29:47.888091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1054/2102] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:29:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:29:47.951959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cdbd60-d7eb-4ee8-9ac1-ca8c158d977e None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:47.952223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cdbd60-d7eb-4ee8-9ac1-ca8c158d977e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:47.952493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cdbd60-d7eb-4ee8-9ac1-ca8c158d977e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:47.972882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d4cdbd60-d7eb-4ee8-9ac1-ca8c158d977e None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:29:47.973997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d4cdbd60-d7eb-4ee8-9ac1-ca8c158d977e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:29:47.981827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1049/2103] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:29:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:29:47.985876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03b8902-d88a-49bb-82a0-0a522b45fe95 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:47.996085 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03b8902-d88a-49bb-82a0-0a522b45fe95 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:48.000742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:48.001328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.001606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.001879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.002533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:29:48.002837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:48.004467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:48.016412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1055/2104] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:29:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:29:48.020635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cdbd60-d7eb-4ee8-9ac1-ca8c158d977e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:48.029603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4cdbd60-d7eb-4ee8-9ac1-ca8c158d977e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:48.034352 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768efdbc-fdde-4c2a-9b66-fc64bb26e0ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:48.034941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768efdbc-fdde-4c2a-9b66-fc64bb26e0ec admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.035256 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768efdbc-fdde-4c2a-9b66-fc64bb26e0ec admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.035526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-768efdbc-fdde-4c2a-9b66-fc64bb26e0ec admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.036149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768efdbc-fdde-4c2a-9b66-fc64bb26e0ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:48.038119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-768efdbc-fdde-4c2a-9b66-fc64bb26e0ec admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:48.141566 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1050/2105] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:29:48 2022] POST /identity/v3/projects => generated 385 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:29:48.145544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:48.155450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed889a7-37bc-4547-b11a-bad42d3d9c87 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:48.210535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88327320-e9a4-4f66-9dc1-79a5d0efe0e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:48.211196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88327320-e9a4-4f66-9dc1-79a5d0efe0e4 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.211479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88327320-e9a4-4f66-9dc1-79a5d0efe0e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.211751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88327320-e9a4-4f66-9dc1-79a5d0efe0e4 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.212420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88327320-e9a4-4f66-9dc1-79a5d0efe0e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:48.214368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88327320-e9a4-4f66-9dc1-79a5d0efe0e4 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:48.311828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1056/2106] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:29:48 2022] POST /identity/v3/users => generated 293 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:29:48.315852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768efdbc-fdde-4c2a-9b66-fc64bb26e0ec admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:48.326704 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-768efdbc-fdde-4c2a-9b66-fc64bb26e0ec admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:48.331915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feecdc83-8a71-42e7-b35e-fb7f27e2e916 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:48.332598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feecdc83-8a71-42e7-b35e-fb7f27e2e916 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.332887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feecdc83-8a71-42e7-b35e-fb7f27e2e916 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.333163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feecdc83-8a71-42e7-b35e-fb7f27e2e916 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.333803 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feecdc83-8a71-42e7-b35e-fb7f27e2e916 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:48.335593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feecdc83-8a71-42e7-b35e-fb7f27e2e916 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:48.345403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1051/2107] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:29:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:29:48.349254 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88327320-e9a4-4f66-9dc1-79a5d0efe0e4 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:48.359780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88327320-e9a4-4f66-9dc1-79a5d0efe0e4 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:48.364368 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff824e7-a33b-4397-a803-7be533155d44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:48.364950 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff824e7-a33b-4397-a803-7be533155d44 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.365227 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff824e7-a33b-4397-a803-7be533155d44 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.365494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff824e7-a33b-4397-a803-7be533155d44 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.366164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff824e7-a33b-4397-a803-7be533155d44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:48.367785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff824e7-a33b-4397-a803-7be533155d44 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:48.378674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1057/2108] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:29:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:29:48.382576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feecdc83-8a71-42e7-b35e-fb7f27e2e916 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:48.392430 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feecdc83-8a71-42e7-b35e-fb7f27e2e916 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:48.397036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9651269-d2d9-4767-bc9b-1afa31fcf343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:48.397672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9651269-d2d9-4767-bc9b-1afa31fcf343 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.397958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9651269-d2d9-4767-bc9b-1afa31fcf343 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.398235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9651269-d2d9-4767-bc9b-1afa31fcf343 admin admin] PATH_INFO: `/v3/projects/7f714627908d49b4b8c5db1fb824ff35/users/ec0c0db97c74469e9a78f810bf789651/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.412570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9651269-d2d9-4767-bc9b-1afa31fcf343 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f714627908d49b4b8c5db1fb824ff35, user_id=ec0c0db97c74469e9a78f810bf789651, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:48.415240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9651269-d2d9-4767-bc9b-1afa31fcf343 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:48.524300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1052/2109] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:29:48 2022] PUT /identity/v3/projects/7f714627908d49b4b8c5db1fb824ff35/users/ec0c0db97c74469e9a78f810bf789651/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:29:48.534049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c4bdd-bce6-4307-8bb1-b577bd33de8e None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.534356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c4bdd-bce6-4307-8bb1-b577bd33de8e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.534625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554c4bdd-bce6-4307-8bb1-b577bd33de8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.564434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-554c4bdd-bce6-4307-8bb1-b577bd33de8e None None] MFA Rules not processed for user `ec0c0db97c74469e9a78f810bf789651`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:29:48.565649 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-554c4bdd-bce6-4307-8bb1-b577bd33de8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:29:48.628481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1058/2110] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:29:48 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:29:48.633056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9651269-d2d9-4767-bc9b-1afa31fcf343 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:48.653691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:29:48.694807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9651269-d2d9-4767-bc9b-1afa31fcf343 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:48.699506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0ec13c-3cbd-488e-ae10-001c0e3d955b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:48.700143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0ec13c-3cbd-488e-ae10-001c0e3d955b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.700419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0ec13c-3cbd-488e-ae10-001c0e3d955b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.700691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0ec13c-3cbd-488e-ae10-001c0e3d955b admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.701310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0ec13c-3cbd-488e-ae10-001c0e3d955b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:48.703223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0ec13c-3cbd-488e-ae10-001c0e3d955b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:48.789994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1053/2111] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:29:48 2022] POST /identity/v3/projects => generated 383 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:29:48.791299 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554c4bdd-bce6-4307-8bb1-b577bd33de8e None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:48.802724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554c4bdd-bce6-4307-8bb1-b577bd33de8e None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:48.814716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:29:48.865976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0569c71-d556-4b72-ba31-51622c22c83c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:48.866579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0569c71-d556-4b72-ba31-51622c22c83c admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:48.866859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0569c71-d556-4b72-ba31-51622c22c83c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:48.867175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0569c71-d556-4b72-ba31-51622c22c83c admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:48.867828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0569c71-d556-4b72-ba31-51622c22c83c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:48.870751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0569c71-d556-4b72-ba31-51622c22c83c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.060049 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1059/2112] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:29:48 2022] POST /identity/v3/users => generated 292 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:29:49.063924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0ec13c-3cbd-488e-ae10-001c0e3d955b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.074183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0ec13c-3cbd-488e-ae10-001c0e3d955b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.078648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0adae31-c278-4fdb-990e-5958dc4f942a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.079532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0adae31-c278-4fdb-990e-5958dc4f942a admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.079811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0adae31-c278-4fdb-990e-5958dc4f942a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.080080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0adae31-c278-4fdb-990e-5958dc4f942a admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.080713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0adae31-c278-4fdb-990e-5958dc4f942a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.082418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0adae31-c278-4fdb-990e-5958dc4f942a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.093323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1054/2113] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:29:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:29:49.097238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0569c71-d556-4b72-ba31-51622c22c83c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.111753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0569c71-d556-4b72-ba31-51622c22c83c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.116481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a00037-4056-4497-a511-a1b76ff6ee50 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.117078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a00037-4056-4497-a511-a1b76ff6ee50 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.117356 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a00037-4056-4497-a511-a1b76ff6ee50 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.117713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a00037-4056-4497-a511-a1b76ff6ee50 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.118565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a00037-4056-4497-a511-a1b76ff6ee50 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.120233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a00037-4056-4497-a511-a1b76ff6ee50 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.130622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1060/2114] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:29:49 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:29:49.134971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0adae31-c278-4fdb-990e-5958dc4f942a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.145588 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0adae31-c278-4fdb-990e-5958dc4f942a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.150121 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753e182-b849-4081-aad9-408acc879c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.150760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753e182-b849-4081-aad9-408acc879c42 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.151721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753e182-b849-4081-aad9-408acc879c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.152005 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f753e182-b849-4081-aad9-408acc879c42 admin admin] PATH_INFO: `/v3/projects/ea7742c1cf054d2494a5c12a4def15a3/users/4cf2ccaee3184f6bbbdacb74fcabcc08/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.166222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f753e182-b849-4081-aad9-408acc879c42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea7742c1cf054d2494a5c12a4def15a3, user_id=4cf2ccaee3184f6bbbdacb74fcabcc08, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.168903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f753e182-b849-4081-aad9-408acc879c42 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.231982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1055/2115] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:29:49 2022] PUT /identity/v3/projects/ea7742c1cf054d2494a5c12a4def15a3/users/4cf2ccaee3184f6bbbdacb74fcabcc08/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:29:49.241718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bef1541-cab2-4e1b-8062-190011c607b4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.242008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bef1541-cab2-4e1b-8062-190011c607b4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.245473 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bef1541-cab2-4e1b-8062-190011c607b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.270314 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-4bef1541-cab2-4e1b-8062-190011c607b4 None None] MFA Rules not processed for user `4cf2ccaee3184f6bbbdacb74fcabcc08`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:29:49.271569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4bef1541-cab2-4e1b-8062-190011c607b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:29:49.335367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1061/2116] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:29:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:29:49.340271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753e182-b849-4081-aad9-408acc879c42 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.400195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f753e182-b849-4081-aad9-408acc879c42 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.405379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bcb912-8560-453c-b618-33fd7feee446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.405974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bcb912-8560-453c-b618-33fd7feee446 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.406260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bcb912-8560-453c-b618-33fd7feee446 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.406533 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bcb912-8560-453c-b618-33fd7feee446 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.407197 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bcb912-8560-453c-b618-33fd7feee446 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.409096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bcb912-8560-453c-b618-33fd7feee446 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.514162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1056/2117] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:29:49 2022] POST /identity/v3/projects => generated 385 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:29:49.518347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bef1541-cab2-4e1b-8062-190011c607b4 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.529037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bef1541-cab2-4e1b-8062-190011c607b4 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.584829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7877683a-ef10-4918-abcc-f91e210b607e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.585478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7877683a-ef10-4918-abcc-f91e210b607e admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.585707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7877683a-ef10-4918-abcc-f91e210b607e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.585984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7877683a-ef10-4918-abcc-f91e210b607e admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.586657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7877683a-ef10-4918-abcc-f91e210b607e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.588659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7877683a-ef10-4918-abcc-f91e210b607e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.665347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1062/2118] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:29:49 2022] POST /identity/v3/users => generated 299 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:29:49.669455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bcb912-8560-453c-b618-33fd7feee446 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.679837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bcb912-8560-453c-b618-33fd7feee446 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.684941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bcc961-45d9-46d3-9c2b-160dd47c0503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.685530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bcc961-45d9-46d3-9c2b-160dd47c0503 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.685808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bcc961-45d9-46d3-9c2b-160dd47c0503 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.686079 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bcc961-45d9-46d3-9c2b-160dd47c0503 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.686715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bcc961-45d9-46d3-9c2b-160dd47c0503 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.688341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bcc961-45d9-46d3-9c2b-160dd47c0503 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.699419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1057/2119] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:29:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:29:49.703517 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7877683a-ef10-4918-abcc-f91e210b607e admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.713874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7877683a-ef10-4918-abcc-f91e210b607e admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.718538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7e994-8d96-4967-a063-5e1b35e74054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.719208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7e994-8d96-4967-a063-5e1b35e74054 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.719492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7e994-8d96-4967-a063-5e1b35e74054 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.719765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b7e994-8d96-4967-a063-5e1b35e74054 admin admin] PATH_INFO: `/v3/projects/e918114ee5b04f2d9fcf86cc392bf6f8/users/5db32b6928e54fd18e87172a1ef44d91/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.731544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:29:49.734432 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7e994-8d96-4967-a063-5e1b35e74054 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e918114ee5b04f2d9fcf86cc392bf6f8, user_id=5db32b6928e54fd18e87172a1ef44d91, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.737122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42b7e994-8d96-4967-a063-5e1b35e74054 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.832620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1063/2120] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:29:49 2022] PUT /identity/v3/projects/e918114ee5b04f2d9fcf86cc392bf6f8/users/5db32b6928e54fd18e87172a1ef44d91/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:29:49.836700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bcc961-45d9-46d3-9c2b-160dd47c0503 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.846768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bcc961-45d9-46d3-9c2b-160dd47c0503 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.887635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:29:49.901897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260fb619-0f2f-42fe-973d-2fd8a6275fb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.902494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260fb619-0f2f-42fe-973d-2fd8a6275fb1 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.902771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260fb619-0f2f-42fe-973d-2fd8a6275fb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.903071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260fb619-0f2f-42fe-973d-2fd8a6275fb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.903704 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260fb619-0f2f-42fe-973d-2fd8a6275fb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.905290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260fb619-0f2f-42fe-973d-2fd8a6275fb1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:49.915070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1058/2121] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:29:49 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:29:49.919337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7e994-8d96-4967-a063-5e1b35e74054 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:29:49.930015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42b7e994-8d96-4967-a063-5e1b35e74054 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:29:49.935419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:29:49.936062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:49.936339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:49.936610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] PATH_INFO: `/v3/projects/e918114ee5b04f2d9fcf86cc392bf6f8/users/5db32b6928e54fd18e87172a1ef44d91/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:49.938546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e918114ee5b04f2d9fcf86cc392bf6f8, user_id=5db32b6928e54fd18e87172a1ef44d91, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:29:49.941224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:29:50.002804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1064/2122] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:29:49 2022] PUT /identity/v3/projects/e918114ee5b04f2d9fcf86cc392bf6f8/users/5db32b6928e54fd18e87172a1ef44d91/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:29:50.013749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50b66cf-31de-4b07-8df7-1b129571682a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:29:50.014037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50b66cf-31de-4b07-8df7-1b129571682a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:29:50.014313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50b66cf-31de-4b07-8df7-1b129571682a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:29:50.042904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c50b66cf-31de-4b07-8df7-1b129571682a None None] MFA Rules not processed for user `5db32b6928e54fd18e87172a1ef44d91`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:29:50.044180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c50b66cf-31de-4b07-8df7-1b129571682a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:29:50.110404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1059/2123] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:29:50 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:29:59.999102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=8,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:30:00.107561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=43 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:30:38.027370 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:30:38.029089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:30:38.113700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-187b9b32-46aa-455f-82f8-58c593eb49d3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:30:38.118533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:30:38.119169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:30:38.119442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:30:38.119709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:30:38.122021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:30:38.134038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:30:38.135635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:30:38.145324 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1065/2124] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:30:38 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:30:44.025894 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50b66cf-31de-4b07-8df7-1b129571682a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:30:44.027582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c50b66cf-31de-4b07-8df7-1b129571682a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:30:44.083920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50b66cf-31de-4b07-8df7-1b129571682a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:30:44.087471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:30:44.088066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:30:44.088343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:30:44.088614 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:30:44.090395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:30:44.161436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:30:44.162875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:30:44.172581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1060/2125] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:30:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:30:44.604371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:30:44.605768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:30:44.617058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672de6c7-d30f-4448-aa13-d46c931e3c61 service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:30:44.620678 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:30:44.621268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:30:44.621544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:30:44.621818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:30:44.623581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] Could not find project: c9922df8fe74462cab223bf2b0d7ce43.: keystone.exception.ProjectNotFound: Could not find project: c9922df8fe74462cab223bf2b0d7ce43. Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Oct 19 22:30:44.636616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 22:30:44.637723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._validate_token_resources() Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 22:30:44.639005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 22:30:44.640078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 22:30:44.641137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: c9922df8fe74462cab223bf2b0d7ce43. Oct 19 22:30:44.641137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 22:30:44.641137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1066/2126] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:30:44 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 34 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Oct 19 22:30:44.882604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:30:44.892050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cada8a1-791c-4f30-b188-2f1e5be59518 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:30:44.895724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5db9095-2c1c-4f74-9a1c-0518d361aa70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:30:44.896323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db9095-2c1c-4f74-9a1c-0518d361aa70 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:30:44.896603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db9095-2c1c-4f74-9a1c-0518d361aa70 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:30:44.896870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5db9095-2c1c-4f74-9a1c-0518d361aa70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:30:44.957968 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5db9095-2c1c-4f74-9a1c-0518d361aa70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:30:44.959456 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5db9095-2c1c-4f74-9a1c-0518d361aa70 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:30:44.968932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1061/2127] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:30:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:30:54.627100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=43 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:30:54.964821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=81 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:33:37.456161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:33:37.457751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:33:37.522916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900638f2-53b0-4764-a18f-fe9153595c0c service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:33:37.527037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:33:37.527632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:33:37.527909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:33:37.528179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:33:37.529895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:33:37.539601 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:33:37.541019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:33:37.548898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1067/2128] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:33:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:33:47.545375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=30 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:34:44.659718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5db9095-2c1c-4f74-9a1c-0518d361aa70 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:34:44.668936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5db9095-2c1c-4f74-9a1c-0518d361aa70 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:34:44.672618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:34:44.673371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:34:44.673658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:34:44.673933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:34:44.675655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:34:44.733931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:34:44.735404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:34:44.744543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1062/2129] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:34:44 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:34:44.786974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:34:44.797142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51446e70-93ff-4df0-8f16-fcdc7318169c service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:34:44.800733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:34:44.801327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:34:44.801608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:34:44.801875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:34:44.803596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:34:44.862233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:34:44.863735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:34:44.872671 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1068/2130] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:34:44 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:34:44.914979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:34:44.924031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07f2272-b663-4044-8601-01dfbdae2f24 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:34:44.927678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:34:44.928268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:34:44.928548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:34:44.928822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:34:44.930543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:34:44.988517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:34:44.989937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:34:44.998887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1063/2131] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:34:44 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:34:45.078739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:34:45.088036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e77ac93-52b9-464e-8815-2d4d77f09671 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:34:45.091670 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:34:45.092257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:34:45.092534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:34:45.092806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:34:45.094512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:34:45.151765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:34:45.153189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:34:45.162187 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1069/2132] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:34:45 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:34:45.203449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:34:45.212735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d69f9172-d05c-4dde-bf28-42c09addd0c5 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:34:45.216345 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:34:45.216935 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:34:45.217211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:34:45.217485 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:34:45.219249 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:34:45.286843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:34:45.288298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:34:45.306117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1064/2133] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:34:45 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:34:55.158081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=56 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:34:55.303489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:35:43.778894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:35:43.794480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f982534-d0c6-4788-b02e-ba46468403da service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:35:43.799726 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:35:43.803357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:35:43.803357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:35:43.803357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:35:43.809718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:35:43.811329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:35:43.821238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1070/2134] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:35:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:35:53.817060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:36.202041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:36.210716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479d1fd5-ce18-4596-bb35-93151c3cae80 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:36.214732 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ae219e-a264-4ce4-bf3d-be6881a3c676 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:36.215771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ae219e-a264-4ce4-bf3d-be6881a3c676 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:36.216048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ae219e-a264-4ce4-bf3d-be6881a3c676 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:36.216319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ae219e-a264-4ce4-bf3d-be6881a3c676 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:36.224938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ae219e-a264-4ce4-bf3d-be6881a3c676 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:36.226457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ae219e-a264-4ce4-bf3d-be6881a3c676 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:36.237023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1065/2135] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:36 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:36.406355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:36.408019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:36.554592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5211f6c5-2f75-41af-b7b8-4f2e07c5c7aa service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:36.558652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c5adff-5399-4b28-9ddc-555b97d21052 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:36.559404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5adff-5399-4b28-9ddc-555b97d21052 service glance] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:36.559747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5adff-5399-4b28-9ddc-555b97d21052 service glance] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:36.560623 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83c5adff-5399-4b28-9ddc-555b97d21052 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:36.562415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-83c5adff-5399-4b28-9ddc-555b97d21052 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application [None req-83c5adff-5399-4b28-9ddc-555b97d21052 service glance] Could not find project: 0b5ba219e7b44b228ed013f830a61572.: keystone.exception.ProjectNotFound: Could not find project: 0b5ba219e7b44b228ed013f830a61572. Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Oct 19 22:36:36.580876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 22:36:36.582117 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self._validate_token_resources() Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 19 22:36:36.583176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.get_or_create( Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application with Lock( Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._enter() Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.creator() Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application created_value = creator( Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 19 22:36:36.584197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 19 22:36:36.585200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: 0b5ba219e7b44b228ed013f830a61572. Oct 19 22:36:36.585200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: ERROR keystone.server.flask.application Oct 19 22:36:36.588270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1071/2136] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:36 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Oct 19 22:36:36.773311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ae219e-a264-4ce4-bf3d-be6881a3c676 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:36.782307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ae219e-a264-4ce4-bf3d-be6881a3c676 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:36.786025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:36.786626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:36.786904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:36.787663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:36.792712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:36.885462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:36.887546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:36.912875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1066/2137] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:37.052009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c5adff-5399-4b28-9ddc-555b97d21052 service glance] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:37.063226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83c5adff-5399-4b28-9ddc-555b97d21052 service glance] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:37.066863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09cb96d-a05a-4be7-a060-84a0a6c4c93b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:37.067484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09cb96d-a05a-4be7-a060-84a0a6c4c93b service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:37.067761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09cb96d-a05a-4be7-a060-84a0a6c4c93b service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:37.068030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b09cb96d-a05a-4be7-a060-84a0a6c4c93b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:37.133824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09cb96d-a05a-4be7-a060-84a0a6c4c93b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:37.135304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b09cb96d-a05a-4be7-a060-84a0a6c4c93b service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:37.148294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1072/2138] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:37 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:37.534766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:37.536864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:37.553473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdf2b9d-153e-4e3d-8194-fbaa46ef9c01 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:37.556006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:37.556339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:37.556661 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:37.556884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:37.558691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:37.577166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:37.578470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:37.590115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1067/2139] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:37 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:39.196778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09cb96d-a05a-4be7-a060-84a0a6c4c93b service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:39.205366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b09cb96d-a05a-4be7-a060-84a0a6c4c93b service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:39.209017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f1a754-2ecf-4c98-8bf3-755d90925df8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:39.209608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f1a754-2ecf-4c98-8bf3-755d90925df8 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:39.209886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f1a754-2ecf-4c98-8bf3-755d90925df8 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:39.210159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f1a754-2ecf-4c98-8bf3-755d90925df8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:39.219753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f1a754-2ecf-4c98-8bf3-755d90925df8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:39.221175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f1a754-2ecf-4c98-8bf3-755d90925df8 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:39.231520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1073/2140] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:39 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:41.518841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:41.520307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:41.535095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da6b4d7-cfa4-41a6-bc5a-9497f78d2d69 service glance] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:41.541376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5b80aa3-dae8-4d7c-899b-e5619a1689c0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:41.546634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b80aa3-dae8-4d7c-899b-e5619a1689c0 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:41.546634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b80aa3-dae8-4d7c-899b-e5619a1689c0 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:41.546634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5b80aa3-dae8-4d7c-899b-e5619a1689c0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:41.546634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b5b80aa3-dae8-4d7c-899b-e5619a1689c0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:41.557281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b80aa3-dae8-4d7c-899b-e5619a1689c0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:41.558916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5b80aa3-dae8-4d7c-899b-e5619a1689c0 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:41.579857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1068/2141] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:42.770005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f1a754-2ecf-4c98-8bf3-755d90925df8 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:42.784752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f1a754-2ecf-4c98-8bf3-755d90925df8 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:42.791508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:42.792216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:42.792566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:42.792944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:42.795937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:42.809198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:42.811002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:42.822020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1074/2142] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:43.088636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04613461-f1d3-45e3-a84c-94a23f97c169 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:43.089474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04613461-f1d3-45e3-a84c-94a23f97c169 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:43.089474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04613461-f1d3-45e3-a84c-94a23f97c169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:43.116183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-04613461-f1d3-45e3-a84c-94a23f97c169 None None] MFA Rules not processed for user `690632c6d7494c22997ac41257ad32ff`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:43.117697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-04613461-f1d3-45e3-a84c-94a23f97c169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:43.128702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1069/2143] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:36:43.479968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:43.481419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:43.492438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a44702-518c-4957-b815-6a3be0a603f6 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:43.495967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:43.497031 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:43.497303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:43.497565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:43.499201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:43.508161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:43.509567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:43.517860 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1075/2144] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:43 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:44.293052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04613461-f1d3-45e3-a84c-94a23f97c169 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:44.306325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04613461-f1d3-45e3-a84c-94a23f97c169 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:44.307868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:44.308369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:44.308651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:44.308958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:44.310692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:44.396333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:44.427730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:44.430245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:44.453483 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1070/2145] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:44.624386 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:44.638188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f8007b-e97b-4897-8b8b-ff2b85136987 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:44.643326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2a821cc-c7c5-4e30-90e1-df8aa27d9282 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:44.643986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a821cc-c7c5-4e30-90e1-df8aa27d9282 service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:44.644325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a821cc-c7c5-4e30-90e1-df8aa27d9282 service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:44.644679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2a821cc-c7c5-4e30-90e1-df8aa27d9282 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:44.646565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d2a821cc-c7c5-4e30-90e1-df8aa27d9282 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:44.688765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a821cc-c7c5-4e30-90e1-df8aa27d9282 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:44.711468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2a821cc-c7c5-4e30-90e1-df8aa27d9282 service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:44.727503 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1076/2146] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:45.581120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:45.593314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79bfca2-dbb4-4d08-9066-1945c395a70c service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:45.600551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:45.601567 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:45.601911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:45.602253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] PATH_INFO: `/v3/users/ec0c0db97c74469e9a78f810bf789651` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:45.603544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec0c0db97c74469e9a78f810bf789651)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:45.606248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:45.841852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec0c0db97c74469e9a78f810bf789651'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:45.842586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec0c0db97c74469e9a78f810bf789651'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:45.865159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec0c0db97c74469e9a78f810bf789651'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:45.873543 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec0c0db97c74469e9a78f810bf789651'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:45.874304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec0c0db97c74469e9a78f810bf789651'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:45.956948 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1071/2147] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:36:45 2022] DELETE /identity/v3/users/ec0c0db97c74469e9a78f810bf789651 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:45.961518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e6e09-0a2b-4797-aca7-293c2c830cf3 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:45.962038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e6e09-0a2b-4797-aca7-293c2c830cf3 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:45.963184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6e6e09-0a2b-4797-aca7-293c2c830cf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:45.987468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-6e6e6e09-0a2b-4797-aca7-293c2c830cf3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:45.988965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6e6e6e09-0a2b-4797-aca7-293c2c830cf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:46.008694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:46.062986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1077/2148] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:36:46.073501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:46.075617 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:46.146106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6640b3-3ed0-4fc7-b4e6-beed9bc03e74 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:46.152497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2fbfd2-1a23-41d6-ba93-916d6888fa58 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:46.153718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2fbfd2-1a23-41d6-ba93-916d6888fa58 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:46.154002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2fbfd2-1a23-41d6-ba93-916d6888fa58 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:46.154275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2fbfd2-1a23-41d6-ba93-916d6888fa58 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:46.170181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2fbfd2-1a23-41d6-ba93-916d6888fa58 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:46.172081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2fbfd2-1a23-41d6-ba93-916d6888fa58 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:46.214735 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1072/2149] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:47.112450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e6e09-0a2b-4797-aca7-293c2c830cf3 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:47.113945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6e6e6e09-0a2b-4797-aca7-293c2c830cf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:47.131673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6e6e09-0a2b-4797-aca7-293c2c830cf3 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:47.137463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:47.139156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:47.139705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:47.139989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] PATH_INFO: `/v3/projects/7f714627908d49b4b8c5db1fb824ff35` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:47.141174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f714627908d49b4b8c5db1fb824ff35)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:47.143191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:47.244489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f714627908d49b4b8c5db1fb824ff35', 'request_id': 'req-9940fa80-be5f-415a-bdf6-a85b9c47fa30'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.325271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f714627908d49b4b8c5db1fb824ff35', 'request_id': 'req-9940fa80-be5f-415a-bdf6-a85b9c47fa30'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.333232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f714627908d49b4b8c5db1fb824ff35', 'request_id': 'req-9940fa80-be5f-415a-bdf6-a85b9c47fa30'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.347793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] The token cache is being invalidate because project 7f714627908d49b4b8c5db1fb824ff35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:36:47.348318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.348984 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.350672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1078/2150] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:36:47 2022] DELETE /identity/v3/projects/7f714627908d49b4b8c5db1fb824ff35 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:47.354611 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2fbfd2-1a23-41d6-ba93-916d6888fa58 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:47.367282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2fbfd2-1a23-41d6-ba93-916d6888fa58 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:47.427502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:47.447950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:47.449028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:47.453224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:47.453224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] PATH_INFO: `/v3/users/4cf2ccaee3184f6bbbdacb74fcabcc08` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:47.453224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4cf2ccaee3184f6bbbdacb74fcabcc08)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:47.454060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:47.682673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4cf2ccaee3184f6bbbdacb74fcabcc08'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.683673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4cf2ccaee3184f6bbbdacb74fcabcc08'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.696413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4cf2ccaee3184f6bbbdacb74fcabcc08'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.704972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4cf2ccaee3184f6bbbdacb74fcabcc08'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.705726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4cf2ccaee3184f6bbbdacb74fcabcc08'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:47.761112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1073/2151] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:36:47 2022] DELETE /identity/v3/users/4cf2ccaee3184f6bbbdacb74fcabcc08 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:48.413531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:48.415287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:48.431352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9940fa80-be5f-415a-bdf6-a85b9c47fa30 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:48.500269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:48.500939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:48.501212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:48.501622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] PATH_INFO: `/v3/projects/ea7742c1cf054d2494a5c12a4def15a3` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:48.502780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea7742c1cf054d2494a5c12a4def15a3)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:48.504750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:48.660898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:48.664916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea7742c1cf054d2494a5c12a4def15a3', 'request_id': 'req-5a197f96-736c-47fe-94d7-be21b08550da'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:48.748079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea7742c1cf054d2494a5c12a4def15a3', 'request_id': 'req-5a197f96-736c-47fe-94d7-be21b08550da'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:48.758874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea7742c1cf054d2494a5c12a4def15a3', 'request_id': 'req-5a197f96-736c-47fe-94d7-be21b08550da'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:48.776581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] The token cache is being invalidate because project ea7742c1cf054d2494a5c12a4def15a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:36:48.778077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:48.778888 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:48.782868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1079/2152] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:36:48 2022] DELETE /identity/v3/projects/ea7742c1cf054d2494a5c12a4def15a3 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:48.788095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:48.790530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:48.812354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f5d8de-a42f-4928-b22d-01171db51106 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:48.926544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:48.927960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:48.928405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:48.928790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] PATH_INFO: `/v3/users/5db32b6928e54fd18e87172a1ef44d91` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:48.931577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5db32b6928e54fd18e87172a1ef44d91)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:48.934872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:49.205372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5db32b6928e54fd18e87172a1ef44d91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.206814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5db32b6928e54fd18e87172a1ef44d91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.223656 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5db32b6928e54fd18e87172a1ef44d91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.233678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5db32b6928e54fd18e87172a1ef44d91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.234675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5db32b6928e54fd18e87172a1ef44d91'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.237755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:49.311516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1074/2153] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:36:48 2022] DELETE /identity/v3/users/5db32b6928e54fd18e87172a1ef44d91 => generated 0 bytes in 525 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:49.426579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:49.428135 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:49.544768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a197f96-736c-47fe-94d7-be21b08550da admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:49.620685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:49.622847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:49.630652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:49.645408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:49.645408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:49.645408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:49.645408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:49.653054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:49.654737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:49.665535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1080/2154] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:49 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:49.692486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ff0af0-e57b-4603-bf4c-a8d9a4e3d0e2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:49.700562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:49.700562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:49.700562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:49.700562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] PATH_INFO: `/v3/users/5ae13baf5df84b588fa65e8dffec4bbf` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:49.700983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ae13baf5df84b588fa65e8dffec4bbf)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:49.702232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:49.937276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ae13baf5df84b588fa65e8dffec4bbf'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.937807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ae13baf5df84b588fa65e8dffec4bbf'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.951654 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ae13baf5df84b588fa65e8dffec4bbf'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.958768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ae13baf5df84b588fa65e8dffec4bbf'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.959497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-0315adfa-1e76-4ad5-bc73-6636ce5da742 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ae13baf5df84b588fa65e8dffec4bbf'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:49.984159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:49.985737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:50.002352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972e6bba-86f7-46f3-9ddf-96699e21f18a service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:50.008266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:50.008904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.009491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.009781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] PATH_INFO: `/v3/projects/e918114ee5b04f2d9fcf86cc392bf6f8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.011060 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e918114ee5b04f2d9fcf86cc392bf6f8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:50.013197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:50.016696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1075/2155] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:36:49 2022] DELETE /identity/v3/users/5ae13baf5df84b588fa65e8dffec4bbf => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:50.114312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e918114ee5b04f2d9fcf86cc392bf6f8', 'request_id': 'req-b19d2356-396e-42dd-b88d-4ceb564924f6'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.165513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e918114ee5b04f2d9fcf86cc392bf6f8', 'request_id': 'req-b19d2356-396e-42dd-b88d-4ceb564924f6'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.184707 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:50.187978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e918114ee5b04f2d9fcf86cc392bf6f8', 'request_id': 'req-b19d2356-396e-42dd-b88d-4ceb564924f6'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.217245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] The token cache is being invalidate because project e918114ee5b04f2d9fcf86cc392bf6f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:36:50.219009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.219009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.222634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1081/2156] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:36:49 2022] DELETE /identity/v3/projects/e918114ee5b04f2d9fcf86cc392bf6f8 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:50.228119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145e5d51-fbd7-4ef8-a519-6d7eeb7a6db2 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.228564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145e5d51-fbd7-4ef8-a519-6d7eeb7a6db2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.228945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145e5d51-fbd7-4ef8-a519-6d7eeb7a6db2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.318432 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:50.320287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:50.339527 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-145e5d51-fbd7-4ef8-a519-6d7eeb7a6db2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:50.343124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-145e5d51-fbd7-4ef8-a519-6d7eeb7a6db2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:50.391191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:INSERT=2,keystone:DELETE=11,keystone:SELECT=87 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:50.403936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19d2356-396e-42dd-b88d-4ceb564924f6 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:50.414001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:50.414376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.414655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.414940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] PATH_INFO: `/v3/projects/2caa701e503b44ddb9fbe09d36a53853` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.431788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] RBAC: Authorizing `identity:delete_project(project_id=2caa701e503b44ddb9fbe09d36a53853)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:50.435202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:50.437647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1076/2157] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:36:50.449028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9d505-9381-40f8-b48b-88139f0fd60b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.449243 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9d505-9381-40f8-b48b-88139f0fd60b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.449600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b9d505-9381-40f8-b48b-88139f0fd60b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.473303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-66b9d505-9381-40f8-b48b-88139f0fd60b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:50.474649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-66b9d505-9381-40f8-b48b-88139f0fd60b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:50.491792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1077/2158] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:50.493074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b9d505-9381-40f8-b48b-88139f0fd60b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:50.505503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b9d505-9381-40f8-b48b-88139f0fd60b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:50.546056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:50.546056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.546056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.546056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.546761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:36:50.546761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:50.546761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:50.546761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1078/2159] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:36:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:36:50.554460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:50.581579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e28248f-cbe4-4332-b868-1914ba4342f2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:50.588339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd51b074-f1f8-4bad-bf92-ae3dac63224a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:50.590612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51b074-f1f8-4bad-bf92-ae3dac63224a admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.591054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51b074-f1f8-4bad-bf92-ae3dac63224a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.591404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd51b074-f1f8-4bad-bf92-ae3dac63224a admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.592594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd51b074-f1f8-4bad-bf92-ae3dac63224a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:50.596951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd51b074-f1f8-4bad-bf92-ae3dac63224a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:50.648465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2caa701e503b44ddb9fbe09d36a53853', 'request_id': 'req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.738403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1079/2160] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:36:50 2022] POST /identity/v3/projects => generated 383 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:36:50.740922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd51b074-f1f8-4bad-bf92-ae3dac63224a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:50.765821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2caa701e503b44ddb9fbe09d36a53853', 'request_id': 'req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.791523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2caa701e503b44ddb9fbe09d36a53853', 'request_id': 'req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.814900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] The token cache is being invalidate because project 2caa701e503b44ddb9fbe09d36a53853 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:36:50.815327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.817352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1d3c717b-f5f5-454f-b43d-13ede4ddb04d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:50.819024 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1082/2161] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:36:50 2022] DELETE /identity/v3/projects/2caa701e503b44ddb9fbe09d36a53853 => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:50.824819 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa772c1-bb4b-490e-b01b-fb4425f12677 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.825104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa772c1-bb4b-490e-b01b-fb4425f12677 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.825785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fa772c1-bb4b-490e-b01b-fb4425f12677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.848300 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1fa772c1-bb4b-490e-b01b-fb4425f12677 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:50.849522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1fa772c1-bb4b-490e-b01b-fb4425f12677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:50.865506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd51b074-f1f8-4bad-bf92-ae3dac63224a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:50.871213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:50.871934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.872142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.872422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.873071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:50.875028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:50.937764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1083/2162] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:50.946843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0149b49-7370-4edb-9397-a65df2a8d2bd None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:50.947198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0149b49-7370-4edb-9397-a65df2a8d2bd None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:50.947431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0149b49-7370-4edb-9397-a65df2a8d2bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:50.968313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c0149b49-7370-4edb-9397-a65df2a8d2bd None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:50.969986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c0149b49-7370-4edb-9397-a65df2a8d2bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:50.984888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1080/2163] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:36:50 2022] POST /identity/v3/users => generated 292 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:36:50.987225 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:50.993935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1084/2164] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:50.997231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:50.998818 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0149b49-7370-4edb-9397-a65df2a8d2bd None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.012703 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0149b49-7370-4edb-9397-a65df2a8d2bd None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.016086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a112d96a-d8e9-4492-8902-37d159fc43cb admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.018327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.018989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.019274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.019540 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.020371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:36:51.020643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.022260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.036351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4c8ea8-c0cb-43fb-b24a-47fb6d7cd997 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.036974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4c8ea8-c0cb-43fb-b24a-47fb6d7cd997 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.037253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4c8ea8-c0cb-43fb-b24a-47fb6d7cd997 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.037523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4c8ea8-c0cb-43fb-b24a-47fb6d7cd997 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.038223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4c8ea8-c0cb-43fb-b24a-47fb6d7cd997 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.039909 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4c8ea8-c0cb-43fb-b24a-47fb6d7cd997 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.044779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1085/2165] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:36:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:36:51.049953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.056003 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1081/2166] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:50 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:36:51.057117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4c8ea8-c0cb-43fb-b24a-47fb6d7cd997 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.068240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dcd5fe-fea5-4445-a72b-13881edfc28f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.069312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4c8ea8-c0cb-43fb-b24a-47fb6d7cd997 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.073245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214acd3f-c989-4583-b13c-d204e0543852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.073848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214acd3f-c989-4583-b13c-d204e0543852 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.073955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deaf977-3b07-4584-8248-d3f67081c73f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.074124 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214acd3f-c989-4583-b13c-d204e0543852 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.074449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214acd3f-c989-4583-b13c-d204e0543852 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.074551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deaf977-3b07-4584-8248-d3f67081c73f admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.074760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deaf977-3b07-4584-8248-d3f67081c73f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.075060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8deaf977-3b07-4584-8248-d3f67081c73f admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.075153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214acd3f-c989-4583-b13c-d204e0543852 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.075699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8deaf977-3b07-4584-8248-d3f67081c73f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.077150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214acd3f-c989-4583-b13c-d204e0543852 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.077467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8deaf977-3b07-4584-8248-d3f67081c73f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.088542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1082/2167] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:51.095335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deaf977-3b07-4584-8248-d3f67081c73f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.117522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8deaf977-3b07-4584-8248-d3f67081c73f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.122800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3650523-5848-42e7-8e3f-31ee89b2b786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.124921 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3650523-5848-42e7-8e3f-31ee89b2b786 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.125203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3650523-5848-42e7-8e3f-31ee89b2b786 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.125477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3650523-5848-42e7-8e3f-31ee89b2b786 admin admin] PATH_INFO: `/v3/projects/465a24314c5241eabd45cabf25715965/users/b2ac1048d47f42f581290b7b2c620be2/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.143212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3650523-5848-42e7-8e3f-31ee89b2b786 admin admin] RBAC: Authorizing `identity:create_grant(project_id=465a24314c5241eabd45cabf25715965, user_id=b2ac1048d47f42f581290b7b2c620be2, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.145945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3650523-5848-42e7-8e3f-31ee89b2b786 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.192037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1086/2168] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/projects => generated 405 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:36:51.193491 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214acd3f-c989-4583-b13c-d204e0543852 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.230455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1083/2169] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:51 2022] PUT /identity/v3/projects/465a24314c5241eabd45cabf25715965/users/b2ac1048d47f42f581290b7b2c620be2/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:51.242937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd3e159-a950-4d4d-83fc-e096fe777daf None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.243327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd3e159-a950-4d4d-83fc-e096fe777daf None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.243691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd3e159-a950-4d4d-83fc-e096fe777daf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.265363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214acd3f-c989-4583-b13c-d204e0543852 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.272438 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2bd3e159-a950-4d4d-83fc-e096fe777daf None None] MFA Rules not processed for user `b2ac1048d47f42f581290b7b2c620be2`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:51.273643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2bd3e159-a950-4d4d-83fc-e096fe777daf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:51.286222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:51.323236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0da8e7f-8c3b-4bf1-8912-8a8deac3e1b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.323790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0da8e7f-8c3b-4bf1-8912-8a8deac3e1b6 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.324065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0da8e7f-8c3b-4bf1-8912-8a8deac3e1b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.324869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0da8e7f-8c3b-4bf1-8912-8a8deac3e1b6 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.325499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0da8e7f-8c3b-4bf1-8912-8a8deac3e1b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.327443 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0da8e7f-8c3b-4bf1-8912-8a8deac3e1b6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.339321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1084/2170] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:51.344142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd3e159-a950-4d4d-83fc-e096fe777daf None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.355383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd3e159-a950-4d4d-83fc-e096fe777daf None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.360017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ffb79a-9fc4-4999-8fb3-b11efe5955d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.360591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ffb79a-9fc4-4999-8fb3-b11efe5955d0 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.360867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ffb79a-9fc4-4999-8fb3-b11efe5955d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.361165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ffb79a-9fc4-4999-8fb3-b11efe5955d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.361785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ffb79a-9fc4-4999-8fb3-b11efe5955d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.363789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ffb79a-9fc4-4999-8fb3-b11efe5955d0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.404766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1087/2171] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/users => generated 303 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:36:51.408591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0da8e7f-8c3b-4bf1-8912-8a8deac3e1b6 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.418053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0da8e7f-8c3b-4bf1-8912-8a8deac3e1b6 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.422683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac7a0c6-e734-429e-9a40-58e02c3ff10f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.423325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac7a0c6-e734-429e-9a40-58e02c3ff10f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.423606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac7a0c6-e734-429e-9a40-58e02c3ff10f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.423934 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac7a0c6-e734-429e-9a40-58e02c3ff10f admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.424580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac7a0c6-e734-429e-9a40-58e02c3ff10f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.426175 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac7a0c6-e734-429e-9a40-58e02c3ff10f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.435960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1088/2172] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:51 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:36:51.439648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac7a0c6-e734-429e-9a40-58e02c3ff10f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.449096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac7a0c6-e734-429e-9a40-58e02c3ff10f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.454248 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95f3678-d725-4c62-a948-944e00921709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.454794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f3678-d725-4c62-a948-944e00921709 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.459506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f3678-d725-4c62-a948-944e00921709 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.459506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95f3678-d725-4c62-a948-944e00921709 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.459506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95f3678-d725-4c62-a948-944e00921709 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.459506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95f3678-d725-4c62-a948-944e00921709 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.468445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1089/2173] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:51 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:51.472231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95f3678-d725-4c62-a948-944e00921709 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.481495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95f3678-d725-4c62-a948-944e00921709 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.488112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdfc73a-e0bf-419b-bdcb-86e0d130b4c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.488813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdfc73a-e0bf-419b-bdcb-86e0d130b4c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.489055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdfc73a-e0bf-419b-bdcb-86e0d130b4c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.489329 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdfc73a-e0bf-419b-bdcb-86e0d130b4c2 admin admin] PATH_INFO: `/v3/projects/bdf9f1b91469433abf076a7912ddbf9a/users/ab234253b422490d90f586329a1cefc3/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.491731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1085/2174] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/projects => generated 385 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:36:51.495435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ffb79a-9fc4-4999-8fb3-b11efe5955d0 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.511645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdfc73a-e0bf-419b-bdcb-86e0d130b4c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdf9f1b91469433abf076a7912ddbf9a, user_id=ab234253b422490d90f586329a1cefc3, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.517056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdfc73a-e0bf-419b-bdcb-86e0d130b4c2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.574134 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ffb79a-9fc4-4999-8fb3-b11efe5955d0 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.579768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954b9389-50bb-4c24-9ecd-00ffc82c2859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.580395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b9389-50bb-4c24-9ecd-00ffc82c2859 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.580665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b9389-50bb-4c24-9ecd-00ffc82c2859 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.580934 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b9389-50bb-4c24-9ecd-00ffc82c2859 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.581610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954b9389-50bb-4c24-9ecd-00ffc82c2859 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.583659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954b9389-50bb-4c24-9ecd-00ffc82c2859 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.594485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1090/2175] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:51 2022] PUT /identity/v3/projects/bdf9f1b91469433abf076a7912ddbf9a/users/ab234253b422490d90f586329a1cefc3/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:51.604592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b43f3-8674-4a89-91f8-1b7aa95ad817 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.604879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b43f3-8674-4a89-91f8-1b7aa95ad817 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.605156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101b43f3-8674-4a89-91f8-1b7aa95ad817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.611000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:51.645467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-101b43f3-8674-4a89-91f8-1b7aa95ad817 None None] MFA Rules not processed for user `ab234253b422490d90f586329a1cefc3`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:51.646713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-101b43f3-8674-4a89-91f8-1b7aa95ad817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:51.682753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1086/2176] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/users => generated 293 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:36:51.687165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954b9389-50bb-4c24-9ecd-00ffc82c2859 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.697155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954b9389-50bb-4c24-9ecd-00ffc82c2859 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.724608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1091/2177] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:51.729869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101b43f3-8674-4a89-91f8-1b7aa95ad817 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.764102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cf9ddb-4606-43b1-8d9d-775eabb49619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.764871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf9ddb-4606-43b1-8d9d-775eabb49619 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.765145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf9ddb-4606-43b1-8d9d-775eabb49619 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.765414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50cf9ddb-4606-43b1-8d9d-775eabb49619 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.766101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cf9ddb-4606-43b1-8d9d-775eabb49619 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.767990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50cf9ddb-4606-43b1-8d9d-775eabb49619 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.782477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1087/2178] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:51 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:51.784167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cf9ddb-4606-43b1-8d9d-775eabb49619 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.794336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50cf9ddb-4606-43b1-8d9d-775eabb49619 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.799988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8000605a-a9b0-4ae2-9ef3-696c117a13b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.800680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8000605a-a9b0-4ae2-9ef3-696c117a13b9 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.801039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8000605a-a9b0-4ae2-9ef3-696c117a13b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.801393 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8000605a-a9b0-4ae2-9ef3-696c117a13b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.801948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101b43f3-8674-4a89-91f8-1b7aa95ad817 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.802175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8000605a-a9b0-4ae2-9ef3-696c117a13b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.803957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8000605a-a9b0-4ae2-9ef3-696c117a13b9 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.808409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d42b1b-d473-4e87-8407-c9540ca987f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.808409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d42b1b-d473-4e87-8407-c9540ca987f2 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.808409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d42b1b-d473-4e87-8407-c9540ca987f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.809516 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d42b1b-d473-4e87-8407-c9540ca987f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.810890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d42b1b-d473-4e87-8407-c9540ca987f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.813262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d42b1b-d473-4e87-8407-c9540ca987f2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.820628 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1088/2179] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:51 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:36:51.821540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8000605a-a9b0-4ae2-9ef3-696c117a13b9 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.832415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8000605a-a9b0-4ae2-9ef3-696c117a13b9 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:51.839560 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37f1348-06d6-449e-8e0f-004dfcd29903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:51.840378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37f1348-06d6-449e-8e0f-004dfcd29903 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.847757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37f1348-06d6-449e-8e0f-004dfcd29903 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.847757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37f1348-06d6-449e-8e0f-004dfcd29903 admin admin] PATH_INFO: `/v3/projects/fc1ce6b33ae749ef81a0f6c1cc6b8357/users/a80bfe9aaf7f4f218abab8dbaf18e1fb/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:51.857618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37f1348-06d6-449e-8e0f-004dfcd29903 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fc1ce6b33ae749ef81a0f6c1cc6b8357, user_id=a80bfe9aaf7f4f218abab8dbaf18e1fb, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:51.860379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37f1348-06d6-449e-8e0f-004dfcd29903 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:51.931829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1092/2180] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/projects => generated 405 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:36:51.936299 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d42b1b-d473-4e87-8407-c9540ca987f2 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:51.965689 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1089/2181] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:51 2022] PUT /identity/v3/projects/fc1ce6b33ae749ef81a0f6c1cc6b8357/users/a80bfe9aaf7f4f218abab8dbaf18e1fb/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:51.977402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755422ab-084f-4c5c-8b39-ad37d7293f4e None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:51.977694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755422ab-084f-4c5c-8b39-ad37d7293f4e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:51.978027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755422ab-084f-4c5c-8b39-ad37d7293f4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.033066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-755422ab-084f-4c5c-8b39-ad37d7293f4e None None] MFA Rules not processed for user `a80bfe9aaf7f4f218abab8dbaf18e1fb`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:52.034349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-755422ab-084f-4c5c-8b39-ad37d7293f4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:52.041556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d42b1b-d473-4e87-8407-c9540ca987f2 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.151590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1090/2182] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:36:52.155231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6185dd24-9181-4cec-937a-fe119285b423 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.156474 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6185dd24-9181-4cec-937a-fe119285b423 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.156760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6185dd24-9181-4cec-937a-fe119285b423 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.157037 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6185dd24-9181-4cec-937a-fe119285b423 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.157765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6185dd24-9181-4cec-937a-fe119285b423 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.159809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6185dd24-9181-4cec-937a-fe119285b423 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.160616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755422ab-084f-4c5c-8b39-ad37d7293f4e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.175789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755422ab-084f-4c5c-8b39-ad37d7293f4e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.180747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1075760-d7bc-4908-8051-8ac80a08e2d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.181501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1075760-d7bc-4908-8051-8ac80a08e2d6 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.181872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1075760-d7bc-4908-8051-8ac80a08e2d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.182273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1075760-d7bc-4908-8051-8ac80a08e2d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.183046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1075760-d7bc-4908-8051-8ac80a08e2d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.185191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1075760-d7bc-4908-8051-8ac80a08e2d6 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.223756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:52.232191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1093/2183] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:36:51 2022] POST /identity/v3/users => generated 309 bytes in 296 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:36:52.237015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6185dd24-9181-4cec-937a-fe119285b423 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.253541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6185dd24-9181-4cec-937a-fe119285b423 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.264645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d713f5bb-6b2a-4e18-9096-2454f108306f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.265288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713f5bb-6b2a-4e18-9096-2454f108306f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.266270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713f5bb-6b2a-4e18-9096-2454f108306f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.266270 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d713f5bb-6b2a-4e18-9096-2454f108306f admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.266662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d713f5bb-6b2a-4e18-9096-2454f108306f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.268339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d713f5bb-6b2a-4e18-9096-2454f108306f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.275010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1091/2184] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:36:52 2022] POST /identity/v3/projects => generated 385 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:36:52.277035 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1075760-d7bc-4908-8051-8ac80a08e2d6 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.284842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1094/2185] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:52.289437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d713f5bb-6b2a-4e18-9096-2454f108306f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.305039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d713f5bb-6b2a-4e18-9096-2454f108306f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.337376 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:52.352444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1075760-d7bc-4908-8051-8ac80a08e2d6 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.358762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145e850f-0ecf-48d1-89e5-9c7cebacc177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.360284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145e850f-0ecf-48d1-89e5-9c7cebacc177 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.360679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145e850f-0ecf-48d1-89e5-9c7cebacc177 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.360961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145e850f-0ecf-48d1-89e5-9c7cebacc177 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.361660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145e850f-0ecf-48d1-89e5-9c7cebacc177 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.363568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145e850f-0ecf-48d1-89e5-9c7cebacc177 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.387378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64139841-cbd7-4273-bb1b-80a716f70af8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.388166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64139841-cbd7-4273-bb1b-80a716f70af8 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.388448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64139841-cbd7-4273-bb1b-80a716f70af8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.388721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64139841-cbd7-4273-bb1b-80a716f70af8 admin admin] PATH_INFO: `/v3/projects/653334b8eb6a4561ada6bc6a3c9f55f5/users/6cd5f0cf56f64c9fa8ae842bd2c99906/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.406487 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64139841-cbd7-4273-bb1b-80a716f70af8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=653334b8eb6a4561ada6bc6a3c9f55f5, user_id=6cd5f0cf56f64c9fa8ae842bd2c99906, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.409651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64139841-cbd7-4273-bb1b-80a716f70af8 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.453754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1092/2186] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:36:52 2022] POST /identity/v3/users => generated 299 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:36:52.454882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145e850f-0ecf-48d1-89e5-9c7cebacc177 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.467888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145e850f-0ecf-48d1-89e5-9c7cebacc177 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.472839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e28bcd-06bc-4525-a976-f160192d29d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.473476 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e28bcd-06bc-4525-a976-f160192d29d5 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.473745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e28bcd-06bc-4525-a976-f160192d29d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.474031 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e28bcd-06bc-4525-a976-f160192d29d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.474677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e28bcd-06bc-4525-a976-f160192d29d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.476321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e28bcd-06bc-4525-a976-f160192d29d5 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.491356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1093/2187] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:52.494086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e28bcd-06bc-4525-a976-f160192d29d5 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.515339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e28bcd-06bc-4525-a976-f160192d29d5 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.515517 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1095/2188] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:52 2022] PUT /identity/v3/projects/653334b8eb6a4561ada6bc6a3c9f55f5/users/6cd5f0cf56f64c9fa8ae842bd2c99906/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:52.522568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64139841-cbd7-4273-bb1b-80a716f70af8 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.576903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca6226-da68-4f9d-ae6a-bdb3bf5228a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.577694 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca6226-da68-4f9d-ae6a-bdb3bf5228a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.578203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca6226-da68-4f9d-ae6a-bdb3bf5228a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.578553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffca6226-da68-4f9d-ae6a-bdb3bf5228a3 admin admin] PATH_INFO: `/v3/projects/b515d39dfd574e7381101d9b78fa1c74/users/5ed76ff1948448a98923679208afbb56/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.597047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffca6226-da68-4f9d-ae6a-bdb3bf5228a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b515d39dfd574e7381101d9b78fa1c74, user_id=5ed76ff1948448a98923679208afbb56, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.600157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffca6226-da68-4f9d-ae6a-bdb3bf5228a3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.604606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64139841-cbd7-4273-bb1b-80a716f70af8 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.611388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe82102-dd28-400a-a374-f3f1030a37f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.612053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe82102-dd28-400a-a374-f3f1030a37f9 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.612286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe82102-dd28-400a-a374-f3f1030a37f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.612555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe82102-dd28-400a-a374-f3f1030a37f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.613223 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe82102-dd28-400a-a374-f3f1030a37f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.614918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe82102-dd28-400a-a374-f3f1030a37f9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.626793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1096/2189] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:36:52.630914 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe82102-dd28-400a-a374-f3f1030a37f9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.640743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe82102-dd28-400a-a374-f3f1030a37f9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.645449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e391f0-a801-4af5-bf83-aa16bfbf54eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.646062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e391f0-a801-4af5-bf83-aa16bfbf54eb admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.646341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e391f0-a801-4af5-bf83-aa16bfbf54eb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.646618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e391f0-a801-4af5-bf83-aa16bfbf54eb admin admin] PATH_INFO: `/v3/projects/653334b8eb6a4561ada6bc6a3c9f55f5/users/6cd5f0cf56f64c9fa8ae842bd2c99906/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.648644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e391f0-a801-4af5-bf83-aa16bfbf54eb admin admin] RBAC: Authorizing `identity:create_grant(project_id=653334b8eb6a4561ada6bc6a3c9f55f5, user_id=6cd5f0cf56f64c9fa8ae842bd2c99906, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.651318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e391f0-a801-4af5-bf83-aa16bfbf54eb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.661842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1097/2190] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:52 2022] PUT /identity/v3/projects/653334b8eb6a4561ada6bc6a3c9f55f5/users/6cd5f0cf56f64c9fa8ae842bd2c99906/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:52.673099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da77e0-8b6e-4be5-b57c-14a09d3b6367 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.673388 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da77e0-8b6e-4be5-b57c-14a09d3b6367 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.675597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93da77e0-8b6e-4be5-b57c-14a09d3b6367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.704880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-93da77e0-8b6e-4be5-b57c-14a09d3b6367 None None] MFA Rules not processed for user `6cd5f0cf56f64c9fa8ae842bd2c99906`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:52.706094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-93da77e0-8b6e-4be5-b57c-14a09d3b6367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:52.707205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1094/2191] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:52 2022] PUT /identity/v3/projects/b515d39dfd574e7381101d9b78fa1c74/users/5ed76ff1948448a98923679208afbb56/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:52.712358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca6226-da68-4f9d-ae6a-bdb3bf5228a3 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.785342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffca6226-da68-4f9d-ae6a-bdb3bf5228a3 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.789706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.790490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.790984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.791437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.792212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.793993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.801984 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:52.809575 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1098/2192] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:52 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:52.809720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1095/2193] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:52 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:52.813133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da77e0-8b6e-4be5-b57c-14a09d3b6367 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.817116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:52.818988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:52.825512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93da77e0-8b6e-4be5-b57c-14a09d3b6367 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.830673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5d9889-a073-40c9-a7bf-f55d2f1aa44f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.831374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5d9889-a073-40c9-a7bf-f55d2f1aa44f admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.831650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5d9889-a073-40c9-a7bf-f55d2f1aa44f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.833174 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5d9889-a073-40c9-a7bf-f55d2f1aa44f admin admin] PATH_INFO: `/v3/projects/b515d39dfd574e7381101d9b78fa1c74/users/5ed76ff1948448a98923679208afbb56/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.834092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5d9889-a073-40c9-a7bf-f55d2f1aa44f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b515d39dfd574e7381101d9b78fa1c74, user_id=5ed76ff1948448a98923679208afbb56, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:52.836791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5d9889-a073-40c9-a7bf-f55d2f1aa44f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:52.845533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:52.850089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1099/2194] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:52 2022] PUT /identity/v3/projects/b515d39dfd574e7381101d9b78fa1c74/users/5ed76ff1948448a98923679208afbb56/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:52.860122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899a964-d62a-4304-bc09-ad5d26e9d795 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.860342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899a964-d62a-4304-bc09-ad5d26e9d795 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.860616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899a964-d62a-4304-bc09-ad5d26e9d795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.888864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e899a964-d62a-4304-bc09-ad5d26e9d795 None None] MFA Rules not processed for user `5ed76ff1948448a98923679208afbb56`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:52.890496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e899a964-d62a-4304-bc09-ad5d26e9d795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:52.890634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac16dea2-53c6-4a88-a756-2151f71fa1ba admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:52.940198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0f51ee-50ca-453c-b292-3bf342536935 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:52.940900 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f51ee-50ca-453c-b292-3bf342536935 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:52.941175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f51ee-50ca-453c-b292-3bf342536935 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:52.941468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb0f51ee-50ca-453c-b292-3bf342536935 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:52.960620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1100/2195] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:52 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:53.016063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0f51ee-50ca-453c-b292-3bf342536935 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:53.017843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb0f51ee-50ca-453c-b292-3bf342536935 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:53.018449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e899a964-d62a-4304-bc09-ad5d26e9d795 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:53.019992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e899a964-d62a-4304-bc09-ad5d26e9d795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:53.030126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1096/2196] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:52 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:36:53.033057 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e899a964-d62a-4304-bc09-ad5d26e9d795 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:53.036904 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2cfb8f-812f-4642-94ae-4a891f50311e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:53.037662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2cfb8f-812f-4642-94ae-4a891f50311e service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:53.037941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2cfb8f-812f-4642-94ae-4a891f50311e service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:53.040100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2cfb8f-812f-4642-94ae-4a891f50311e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:53.116705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2cfb8f-812f-4642-94ae-4a891f50311e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:53.118283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2cfb8f-812f-4642-94ae-4a891f50311e service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:53.128890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1101/2197] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:53 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:36:53.181529 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0f51ee-50ca-453c-b292-3bf342536935 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:53.215672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:53.267538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb0f51ee-50ca-453c-b292-3bf342536935 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:53.272334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:53.273043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:53.273400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:53.273865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] PATH_INFO: `/v3/users/ab234253b422490d90f586329a1cefc3` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:53.275108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ab234253b422490d90f586329a1cefc3)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:53.277153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:53.318829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:53.496838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ab234253b422490d90f586329a1cefc3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:53.497500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ab234253b422490d90f586329a1cefc3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:53.511026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ab234253b422490d90f586329a1cefc3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:53.519397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ab234253b422490d90f586329a1cefc3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:53.520308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ab234253b422490d90f586329a1cefc3'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:53.635336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1097/2198] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:36:53 2022] DELETE /identity/v3/users/ab234253b422490d90f586329a1cefc3 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:53.639810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec9fa5e-5d71-4508-9929-8f736e28b1ca None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:53.640723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec9fa5e-5d71-4508-9929-8f736e28b1ca None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:53.641285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec9fa5e-5d71-4508-9929-8f736e28b1ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:53.663645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3ec9fa5e-5d71-4508-9929-8f736e28b1ca None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:53.665079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3ec9fa5e-5d71-4508-9929-8f736e28b1ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:53.738825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1102/2199] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:53.751821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:53.754235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:53.827515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ddf6dde-7523-45d6-9324-21f71202aaa1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:53.831128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5f43f6-8a2a-404b-9baa-b17d7044acd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:53.831741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f43f6-8a2a-404b-9baa-b17d7044acd0 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:53.832026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f43f6-8a2a-404b-9baa-b17d7044acd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:53.832300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f43f6-8a2a-404b-9baa-b17d7044acd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:53.841943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5f43f6-8a2a-404b-9baa-b17d7044acd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:53.843520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5f43f6-8a2a-404b-9baa-b17d7044acd0 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:53.881113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:53.881627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1098/2200] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:54.236838 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:54.321824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec9fa5e-5d71-4508-9929-8f736e28b1ca None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:54.323358 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3ec9fa5e-5d71-4508-9929-8f736e28b1ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:54.336010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec9fa5e-5d71-4508-9929-8f736e28b1ca None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:54.348286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:54.349039 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:54.349385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:54.349738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] PATH_INFO: `/v3/projects/bdf9f1b91469433abf076a7912ddbf9a` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:54.351575 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdf9f1b91469433abf076a7912ddbf9a)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:54.353599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:54.457268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdf9f1b91469433abf076a7912ddbf9a', 'request_id': 'req-e2b054c1-b780-4d0e-8d35-8123212c5a05'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.507427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdf9f1b91469433abf076a7912ddbf9a', 'request_id': 'req-e2b054c1-b780-4d0e-8d35-8123212c5a05'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.513058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdf9f1b91469433abf076a7912ddbf9a', 'request_id': 'req-e2b054c1-b780-4d0e-8d35-8123212c5a05'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.524663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] The token cache is being invalidate because project bdf9f1b91469433abf076a7912ddbf9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:36:54.524946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.525632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.527238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1103/2201] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:36:54 2022] DELETE /identity/v3/projects/bdf9f1b91469433abf076a7912ddbf9a => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:54.531525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5f43f6-8a2a-404b-9baa-b17d7044acd0 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:54.541486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5f43f6-8a2a-404b-9baa-b17d7044acd0 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:54.597664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:54.598314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:54.598797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:54.599248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] PATH_INFO: `/v3/users/6cd5f0cf56f64c9fa8ae842bd2c99906` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:54.600739 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6cd5f0cf56f64c9fa8ae842bd2c99906)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:54.602672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:54.805019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6cd5f0cf56f64c9fa8ae842bd2c99906'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.805828 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6cd5f0cf56f64c9fa8ae842bd2c99906'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.818740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6cd5f0cf56f64c9fa8ae842bd2c99906'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.829003 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6cd5f0cf56f64c9fa8ae842bd2c99906'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.829736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6cd5f0cf56f64c9fa8ae842bd2c99906'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:54.925591 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1099/2202] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:36:54 2022] DELETE /identity/v3/users/6cd5f0cf56f64c9fa8ae842bd2c99906 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:55.332062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:55.334170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:55.342342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:55.343854 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:55.412910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b054c1-b780-4d0e-8d35-8123212c5a05 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:55.417397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995103fa-5414-4a3c-a34b-88d24a7e3e33 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:55.421555 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:55.422411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:55.422896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:55.423266 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:55.425252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:55.432889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:55.500321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ebed1f-1da1-47a1-a878-f5e084d4efd8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:55.500998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ebed1f-1da1-47a1-a878-f5e084d4efd8 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:55.500998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ebed1f-1da1-47a1-a878-f5e084d4efd8 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:55.501559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ebed1f-1da1-47a1-a878-f5e084d4efd8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:55.504419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d6ebed1f-1da1-47a1-a878-f5e084d4efd8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:55.506603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:55.508417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:55.521014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1100/2203] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:55 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:36:55.522028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:55.523483 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:55.548131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:55.612120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e23e43a-804c-4c03-815d-f5934d1fa057 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:55.619592 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:55.620724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:55.622125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:55.622888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] PATH_INFO: `/v3/projects/653334b8eb6a4561ada6bc6a3c9f55f5` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:55.625195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] RBAC: Authorizing `identity:delete_project(project_id=653334b8eb6a4561ada6bc6a3c9f55f5)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:55.627615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:55.641164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ebed1f-1da1-47a1-a878-f5e084d4efd8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:55.642690 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ebed1f-1da1-47a1-a878-f5e084d4efd8 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:55.660366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1104/2204] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:55.694981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddaf00c-3da0-4b1e-bc6a-25798be2586c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:55.695333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddaf00c-3da0-4b1e-bc6a-25798be2586c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:55.695609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dddaf00c-3da0-4b1e-bc6a-25798be2586c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:55.770347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-dddaf00c-3da0-4b1e-bc6a-25798be2586c None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:55.772379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dddaf00c-3da0-4b1e-bc6a-25798be2586c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:55.778971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '653334b8eb6a4561ada6bc6a3c9f55f5', 'request_id': 'req-15b760e7-63be-4245-a5f7-12bcbda69c90'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:55.785997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '653334b8eb6a4561ada6bc6a3c9f55f5', 'request_id': 'req-15b760e7-63be-4245-a5f7-12bcbda69c90'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:55.791855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '653334b8eb6a4561ada6bc6a3c9f55f5', 'request_id': 'req-15b760e7-63be-4245-a5f7-12bcbda69c90'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:55.868990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1105/2205] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:36:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:55.875055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddaf00c-3da0-4b1e-bc6a-25798be2586c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:55.876495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dddaf00c-3da0-4b1e-bc6a-25798be2586c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:55.906268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] The token cache is being invalidate because project 653334b8eb6a4561ada6bc6a3c9f55f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:36:55.906654 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:55.907440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-15b760e7-63be-4245-a5f7-12bcbda69c90 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:55.909744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1101/2206] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:36:55 2022] DELETE /identity/v3/projects/653334b8eb6a4561ada6bc6a3c9f55f5 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:55.915180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d07e7e-b42a-48d9-969e-f08c94842ac0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:55.915468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d07e7e-b42a-48d9-969e-f08c94842ac0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:55.915746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d07e7e-b42a-48d9-969e-f08c94842ac0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:55.956642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-58d07e7e-b42a-48d9-969e-f08c94842ac0 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:55.958277 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-58d07e7e-b42a-48d9-969e-f08c94842ac0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:55.995137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dddaf00c-3da0-4b1e-bc6a-25798be2586c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:56.002275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:56.003251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.003583 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.003789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.005566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:56.021537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:56.023506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:56.035908 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1106/2207] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:55 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:56.044770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:56.047366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:56.068537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1102/2208] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:36:56.076709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6749582-f862-461b-885d-b617ecf07fc3 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.077273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6749582-f862-461b-885d-b617ecf07fc3 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.077787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6749582-f862-461b-885d-b617ecf07fc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.114942 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a6749582-f862-461b-885d-b617ecf07fc3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:56.119545 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a6749582-f862-461b-885d-b617ecf07fc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:56.139970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1103/2209] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:56.144804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6749582-f862-461b-885d-b617ecf07fc3 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:56.155293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6749582-f862-461b-885d-b617ecf07fc3 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:56.159911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:56.160525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.160838 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.161111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.161787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:36:56.162119 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:56.163809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:56.176918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1104/2210] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:36:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:36:56.180924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:56.196378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cfaa61-344b-4b60-bec2-4f0fd73bd1e8 service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:56.200406 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb1364c-3400-4220-bc37-842ee1a60842 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:56.212213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eee9420-522f-4dc8-b3f4-86498c9ebee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:56.212706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eee9420-522f-4dc8-b3f4-86498c9ebee2 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.213390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eee9420-522f-4dc8-b3f4-86498c9ebee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.213390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eee9420-522f-4dc8-b3f4-86498c9ebee2 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.213945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eee9420-522f-4dc8-b3f4-86498c9ebee2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:56.216206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eee9420-522f-4dc8-b3f4-86498c9ebee2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:56.219898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:56.220886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.221412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.221861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.229482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:56.252928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:56.254945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:56.287023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1107/2211] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:36:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:36:56.329917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1105/2212] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:36:56 2022] POST /identity/v3/projects => generated 365 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:36:56.341241 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:56.387527 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ede757-8469-4f15-a181-fb1b14a2d91a service swift] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:56.525791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:56.534301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230e7b6d-9b3e-4f7d-9e3f-d33acc55cb83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:56.535116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e7b6d-9b3e-4f7d-9e3f-d33acc55cb83 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.535471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e7b6d-9b3e-4f7d-9e3f-d33acc55cb83 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.535878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230e7b6d-9b3e-4f7d-9e3f-d33acc55cb83 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.536755 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230e7b6d-9b3e-4f7d-9e3f-d33acc55cb83 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:56.539268 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230e7b6d-9b3e-4f7d-9e3f-d33acc55cb83 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:56.613113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1108/2213] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:36:56 2022] POST /identity/v3/users => generated 283 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:36:56.617095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eee9420-522f-4dc8-b3f4-86498c9ebee2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:56.649187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eee9420-522f-4dc8-b3f4-86498c9ebee2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:56.655608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e34dce-d28c-4fec-8ad7-73fb23b0de4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:56.656481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e34dce-d28c-4fec-8ad7-73fb23b0de4d admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.656913 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e34dce-d28c-4fec-8ad7-73fb23b0de4d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.657557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e34dce-d28c-4fec-8ad7-73fb23b0de4d admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.658402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e34dce-d28c-4fec-8ad7-73fb23b0de4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:56.660241 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9e34dce-d28c-4fec-8ad7-73fb23b0de4d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:56.676657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230e7b6d-9b3e-4f7d-9e3f-d33acc55cb83 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:56.677090 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1106/2214] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:56 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:36:56.690837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230e7b6d-9b3e-4f7d-9e3f-d33acc55cb83 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:56.698609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72bbc7dd-f81a-4441-a6f5-1ae43208863f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:56.699825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bbc7dd-f81a-4441-a6f5-1ae43208863f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.702144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bbc7dd-f81a-4441-a6f5-1ae43208863f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.702537 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72bbc7dd-f81a-4441-a6f5-1ae43208863f admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.703968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bbc7dd-f81a-4441-a6f5-1ae43208863f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:56.706587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72bbc7dd-f81a-4441-a6f5-1ae43208863f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:56.725428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1109/2215] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:56 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:56.729779 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e34dce-d28c-4fec-8ad7-73fb23b0de4d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:56.747166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e34dce-d28c-4fec-8ad7-73fb23b0de4d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:56.758421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:56.758927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.759371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.759658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] PATH_INFO: `/v3/projects/fbe38f948f7942c4937fca763dbc1959/users/c807be8622c94fdb9c9e3688ed2e72db/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:56.782902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fbe38f948f7942c4937fca763dbc1959, user_id=c807be8622c94fdb9c9e3688ed2e72db, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:56.785563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:56.950378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:56.954329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1107/2216] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:56 2022] PUT /identity/v3/projects/fbe38f948f7942c4937fca763dbc1959/users/c807be8622c94fdb9c9e3688ed2e72db/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:56.964534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74eb989-93c6-4399-abc7-fbab75e4a6b2 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:56.964826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74eb989-93c6-4399-abc7-fbab75e4a6b2 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:56.965624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74eb989-93c6-4399-abc7-fbab75e4a6b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:57.014706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e74eb989-93c6-4399-abc7-fbab75e4a6b2 None None] MFA Rules not processed for user `c807be8622c94fdb9c9e3688ed2e72db`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:57.016007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e74eb989-93c6-4399-abc7-fbab75e4a6b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:57.030268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:57.032493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:57.142242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1110/2217] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:56 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:36:57.144624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74eb989-93c6-4399-abc7-fbab75e4a6b2 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:57.160004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c529311-ce77-48e0-9927-884bdbf14fa4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:57.165342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:57.166502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:57.166933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:57.167247 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] PATH_INFO: `/v3/users/7ea5afc217f843389fe051b42616e09d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:57.169757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] RBAC: Authorizing `identity:delete_user(user_id=7ea5afc217f843389fe051b42616e09d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:57.172669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:57.256640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74eb989-93c6-4399-abc7-fbab75e4a6b2 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:57.265103 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87019a6-878e-4db1-9632-ac36e1ee37d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:57.267036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87019a6-878e-4db1-9632-ac36e1ee37d5 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:57.267036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87019a6-878e-4db1-9632-ac36e1ee37d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:57.267036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87019a6-878e-4db1-9632-ac36e1ee37d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:57.267036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87019a6-878e-4db1-9632-ac36e1ee37d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:57.270291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87019a6-878e-4db1-9632-ac36e1ee37d5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:57.413272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1111/2218] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:36:57 2022] POST /identity/v3/projects => generated 365 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:36:57.414305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87019a6-878e-4db1-9632-ac36e1ee37d5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:57.442887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7ea5afc217f843389fe051b42616e09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:57.451995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7ea5afc217f843389fe051b42616e09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:57.519927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7ea5afc217f843389fe051b42616e09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:57.537114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7ea5afc217f843389fe051b42616e09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:57.539037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7ea5afc217f843389fe051b42616e09d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:57.561004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:57.566136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87019a6-878e-4db1-9632-ac36e1ee37d5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:57.576077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:57.576077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:57.576077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:57.576077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:57.576500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:57.578089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:57.619321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1108/2219] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:36:57 2022] DELETE /identity/v3/users/7ea5afc217f843389fe051b42616e09d => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:57.620940 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:57.626384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:57.727081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1112/2220] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:36:57 2022] POST /identity/v3/users => generated 289 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:36:57.731478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:57.734061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae18864e-bbe1-46b0-9366-5642af99d39f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:57.734262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:57.751755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:57.752633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:57.755161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0b0c61-a49b-49be-b677-ac9f75a95562 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:57.757607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:57.757961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:57.761410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:57.892478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:57.893292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:57.894479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:57.894988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:57.895948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:57.897812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:57.904220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:57.906410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:57.930188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1109/2221] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:57 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:57.937501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1113/2222] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:57 2022] GET /identity/v3/roles => generated 1420 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:57.944049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:57.978717 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15c26b9-833f-4a9b-8a77-23e3ce70c083 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:57.996649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290701f1-1911-49fd-977a-a2388fc79a28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:57.997468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290701f1-1911-49fd-977a-a2388fc79a28 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:57.997818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290701f1-1911-49fd-977a-a2388fc79a28 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:57.998201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290701f1-1911-49fd-977a-a2388fc79a28 admin admin] PATH_INFO: `/v3/projects/85e80f6957e04b2d91358a8456d2d9f1/users/0306b9f5d8d440689407fa2ba0987e6f/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:58.027321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290701f1-1911-49fd-977a-a2388fc79a28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85e80f6957e04b2d91358a8456d2d9f1, user_id=0306b9f5d8d440689407fa2ba0987e6f, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:58.031126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290701f1-1911-49fd-977a-a2388fc79a28 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:58.037644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:58.039685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:58.057212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5e2a64-9469-43b2-8c38-1c983d5927c9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:58.064255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:58.068769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:58.069122 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:58.069475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] PATH_INFO: `/v3/projects/6ec4ff60137544109a46d3a5810251f5` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:58.070771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ec4ff60137544109a46d3a5810251f5)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:58.091192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:58.115875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1110/2223] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:57 2022] PUT /identity/v3/projects/85e80f6957e04b2d91358a8456d2d9f1/users/0306b9f5d8d440689407fa2ba0987e6f/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:58.119779 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290701f1-1911-49fd-977a-a2388fc79a28 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:58.192864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:58.259881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290701f1-1911-49fd-977a-a2388fc79a28 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:58.294663 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ec4ff60137544109a46d3a5810251f5', 'request_id': 'req-d785fc36-4d9e-4a58-859e-034f407c3291'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:58.357872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc3557-3fb1-4fdd-a44d-8a4a11b41b1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:58.358471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc3557-3fb1-4fdd-a44d-8a4a11b41b1d admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:58.358751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc3557-3fb1-4fdd-a44d-8a4a11b41b1d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:58.359197 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebc3557-3fb1-4fdd-a44d-8a4a11b41b1d admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:58.359903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebc3557-3fb1-4fdd-a44d-8a4a11b41b1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:58.361599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ebc3557-3fb1-4fdd-a44d-8a4a11b41b1d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:58.373757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1111/2224] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:36:58 2022] GET /identity/v3/roles => generated 1420 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:36:58.378133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc3557-3fb1-4fdd-a44d-8a4a11b41b1d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:58.390285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ebc3557-3fb1-4fdd-a44d-8a4a11b41b1d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:58.399508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ec4ff60137544109a46d3a5810251f5', 'request_id': 'req-d785fc36-4d9e-4a58-859e-034f407c3291'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:58.401966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:58.405916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:58.406651 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ec4ff60137544109a46d3a5810251f5', 'request_id': 'req-d785fc36-4d9e-4a58-859e-034f407c3291'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:58.416772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:58.416772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] PATH_INFO: `/v3/projects/85e80f6957e04b2d91358a8456d2d9f1/users/0306b9f5d8d440689407fa2ba0987e6f/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:58.416772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85e80f6957e04b2d91358a8456d2d9f1, user_id=0306b9f5d8d440689407fa2ba0987e6f, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:58.418920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:58.423317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] The token cache is being invalidate because project 6ec4ff60137544109a46d3a5810251f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:36:58.424148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:58.435640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d785fc36-4d9e-4a58-859e-034f407c3291 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:36:58.438498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1114/2226] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:36:58 2022] DELETE /identity/v3/projects/6ec4ff60137544109a46d3a5810251f5 => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:36:58.439148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1112/2226] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:36:58 2022] PUT /identity/v3/projects/85e80f6957e04b2d91358a8456d2d9f1/users/0306b9f5d8d440689407fa2ba0987e6f/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:36:58.448488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db2855-baea-4901-83d2-1bf6fc5deb13 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:58.448488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db2855-baea-4901-83d2-1bf6fc5deb13 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:58.448488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db2855-baea-4901-83d2-1bf6fc5deb13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:58.497954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:58.511458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-58db2855-baea-4901-83d2-1bf6fc5deb13 None None] MFA Rules not processed for user `0306b9f5d8d440689407fa2ba0987e6f`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:36:58.513332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-58db2855-baea-4901-83d2-1bf6fc5deb13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:58.635922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1115/2227] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:36:58 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:36:58.649874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:58.651997 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:58.743084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e4264b-b917-4bba-bab6-c4ce280769a6 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:58.748484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:58.755727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:58.755962 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:58.759216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:58.774596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:58.776616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:58.789841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1113/2228] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:58 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:36:59.042078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db2855-baea-4901-83d2-1bf6fc5deb13 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:36:59.061247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db2855-baea-4901-83d2-1bf6fc5deb13 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:36:59.087170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ecfdab-981c-4761-b73f-f71682a9e92f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:36:59.092059 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ecfdab-981c-4761-b73f-f71682a9e92f service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:36:59.092760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ecfdab-981c-4761-b73f-f71682a9e92f service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:36:59.095963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ecfdab-981c-4761-b73f-f71682a9e92f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:36:59.103017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e3ecfdab-981c-4761-b73f-f71682a9e92f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:36:59.159957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=2,keystone:DELETE=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:59.212574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ecfdab-981c-4761-b73f-f71682a9e92f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:36:59.214055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ecfdab-981c-4761-b73f-f71682a9e92f service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:36:59.219213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:36:59.225541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1116/2229] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:36:59 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:03.133932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:03.134369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:03.269440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f381112a-9cb1-4992-a83a-657f300430f6 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:03.280977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:03.281626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:03.281903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:03.282181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] PATH_INFO: `/v3/users/b2ac1048d47f42f581290b7b2c620be2` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:03.284907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2ac1048d47f42f581290b7b2c620be2)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:03.286853 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:03.537835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:03.558396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2ac1048d47f42f581290b7b2c620be2'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:03.559782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2ac1048d47f42f581290b7b2c620be2'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:03.580500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2ac1048d47f42f581290b7b2c620be2'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:03.594599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2ac1048d47f42f581290b7b2c620be2'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:03.599370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2ac1048d47f42f581290b7b2c620be2'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:03.682858 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1114/2230] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:37:03 2022] DELETE /identity/v3/users/b2ac1048d47f42f581290b7b2c620be2 => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:03.687489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ede471-241f-469e-b958-b04bdbb0fe48 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:03.688022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ede471-241f-469e-b958-b04bdbb0fe48 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:03.688128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ede471-241f-469e-b958-b04bdbb0fe48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:03.732872 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e0ede471-241f-469e-b958-b04bdbb0fe48 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:03.734820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e0ede471-241f-469e-b958-b04bdbb0fe48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:03.859858 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1117/2231] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:03.870199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:03.872173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:03.969978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e91779-b2cd-4fde-9ec3-abc7f8c612cc admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:03.974357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:03.976454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:03.977905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:03.978262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:03.992553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:03.994066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:04.005932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1115/2232] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:04.623010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13fa8f-5846-4e67-b69c-3596834e456f None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:04.623327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13fa8f-5846-4e67-b69c-3596834e456f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:04.623600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13fa8f-5846-4e67-b69c-3596834e456f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:04.666472 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0c13fa8f-5846-4e67-b69c-3596834e456f None None] MFA Rules not processed for user `57b6c2a1a4a248158c5abbabe61ad634`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:04.667717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0c13fa8f-5846-4e67-b69c-3596834e456f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:04.733635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:04.735752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:04.751594 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18298a33-3575-4f6a-8788-bc557297ef07 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:04.754612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1118/2233] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:04.757475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:04.758115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:04.758392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:04.758666 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] PATH_INFO: `/v3/projects/465a24314c5241eabd45cabf25715965` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:04.760075 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] RBAC: Authorizing `identity:delete_project(project_id=465a24314c5241eabd45cabf25715965)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:04.762758 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:04.920213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '465a24314c5241eabd45cabf25715965', 'request_id': 'req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:04.936685 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '465a24314c5241eabd45cabf25715965', 'request_id': 'req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:04.943188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '465a24314c5241eabd45cabf25715965', 'request_id': 'req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:04.986143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c13fa8f-5846-4e67-b69c-3596834e456f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:04.997264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c13fa8f-5846-4e67-b69c-3596834e456f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:05.052683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] The token cache is being invalidate because project 465a24314c5241eabd45cabf25715965 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:37:05.053280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:05.054662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:05.056728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1116/2234] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:37:04 2022] DELETE /identity/v3/projects/465a24314c5241eabd45cabf25715965 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:05.061986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:05.067590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:05.068545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:05.069165 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:05.069535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:05.069810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:05.072373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:05.154675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:05.158188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:05.192866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d2164b-e20a-428c-abe8-5c7af9cb4c6a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:05.196922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:05.198153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:05.198794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:05.199348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:05.199634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:05.199916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] PATH_INFO: `/v3/users/a80bfe9aaf7f4f218abab8dbaf18e1fb` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:05.201475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a80bfe9aaf7f4f218abab8dbaf18e1fb)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:05.203221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:05.237966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1119/2235] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:04 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:05.421286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a80bfe9aaf7f4f218abab8dbaf18e1fb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:05.421954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a80bfe9aaf7f4f218abab8dbaf18e1fb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:05.462007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a80bfe9aaf7f4f218abab8dbaf18e1fb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:05.477875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a80bfe9aaf7f4f218abab8dbaf18e1fb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:05.479504 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a80bfe9aaf7f4f218abab8dbaf18e1fb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:05.603716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1117/2236] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:37:05 2022] DELETE /identity/v3/users/a80bfe9aaf7f4f218abab8dbaf18e1fb => generated 0 bytes in 542 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:06.213918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:06.237889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436d6f16-123f-4104-9d27-7831b5278c47 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:06.370529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:06.371283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:06.371566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:06.371837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] PATH_INFO: `/v3/projects/fc1ce6b33ae749ef81a0f6c1cc6b8357` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:06.373151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fc1ce6b33ae749ef81a0f6c1cc6b8357)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:06.381374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:06.577361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fc1ce6b33ae749ef81a0f6c1cc6b8357', 'request_id': 'req-b9c217c7-2879-41b2-919b-9d5ddc724441'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:06.668924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fc1ce6b33ae749ef81a0f6c1cc6b8357', 'request_id': 'req-b9c217c7-2879-41b2-919b-9d5ddc724441'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:06.678403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fc1ce6b33ae749ef81a0f6c1cc6b8357', 'request_id': 'req-b9c217c7-2879-41b2-919b-9d5ddc724441'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:06.693418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] The token cache is being invalidate because project fc1ce6b33ae749ef81a0f6c1cc6b8357 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:37:06.694090 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:06.695503 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:06.697312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1120/2237] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:37:06 2022] DELETE /identity/v3/projects/fc1ce6b33ae749ef81a0f6c1cc6b8357 => generated 0 bytes in 488 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:06.701490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:06.703029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:06.720117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e01c95-83e3-4df8-bd3c-bb0f6a2493f1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:06.821248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:06.821898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:06.822181 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:06.822605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] PATH_INFO: `/v3/users/5ed76ff1948448a98923679208afbb56` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:06.823799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ed76ff1948448a98923679208afbb56)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:06.825777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:07.012473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ed76ff1948448a98923679208afbb56'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:07.013673 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ed76ff1948448a98923679208afbb56'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:07.030439 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ed76ff1948448a98923679208afbb56'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:07.031933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=80,keystone:DELETE=19,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:07.039721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ed76ff1948448a98923679208afbb56'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:07.040610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b3c04b99-8502-42aa-b7f5-ed06848b34bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ed76ff1948448a98923679208afbb56'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:07.106342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1118/2238] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:37:06 2022] DELETE /identity/v3/users/5ed76ff1948448a98923679208afbb56 => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:07.797512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:07.799345 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:07.814652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c217c7-2879-41b2-919b-9d5ddc724441 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:07.935941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:07.937020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:07.937506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:07.938011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] PATH_INFO: `/v3/projects/b515d39dfd574e7381101d9b78fa1c74` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:07.939516 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b515d39dfd574e7381101d9b78fa1c74)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:07.942775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:07.945007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:08.134145 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b515d39dfd574e7381101d9b78fa1c74', 'request_id': 'req-98bbe81c-c060-45e5-8941-4d46bce05065'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:08.217546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b515d39dfd574e7381101d9b78fa1c74', 'request_id': 'req-98bbe81c-c060-45e5-8941-4d46bce05065'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:08.224937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b515d39dfd574e7381101d9b78fa1c74', 'request_id': 'req-98bbe81c-c060-45e5-8941-4d46bce05065'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:08.248169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] The token cache is being invalidate because project b515d39dfd574e7381101d9b78fa1c74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:37:08.248418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:08.249137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-98bbe81c-c060-45e5-8941-4d46bce05065 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:08.251219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1121/2239] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:37:07 2022] DELETE /identity/v3/projects/b515d39dfd574e7381101d9b78fa1c74 => generated 0 bytes in 460 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:08.257103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963a033-4e70-45b4-9c41-bd60c2f958e1 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:08.257651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963a033-4e70-45b4-9c41-bd60c2f958e1 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:08.257651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c963a033-4e70-45b4-9c41-bd60c2f958e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:08.281051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c963a033-4e70-45b4-9c41-bd60c2f958e1 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:08.282907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c963a033-4e70-45b4-9c41-bd60c2f958e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:08.376047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1119/2240] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:08.384851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8cdf54-a9cb-4e48-beea-964544065180 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:08.384851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8cdf54-a9cb-4e48-beea-964544065180 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:08.391876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8cdf54-a9cb-4e48-beea-964544065180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:08.434172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3d8cdf54-a9cb-4e48-beea-964544065180 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:08.435470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3d8cdf54-a9cb-4e48-beea-964544065180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:08.461765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c963a033-4e70-45b4-9c41-bd60c2f958e1 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:08.462807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1122/2241] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:08.506778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c963a033-4e70-45b4-9c41-bd60c2f958e1 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:08.518324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:08.519727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:08.520165 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:08.520618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:08.521455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:37:08.523751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:08.525628 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:08.564168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1120/2242] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:37:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:37:08.568810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8cdf54-a9cb-4e48-beea-964544065180 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:08.588359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8cdf54-a9cb-4e48-beea-964544065180 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:08.605595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d918425c-ba9f-49c8-a636-761e5d2e0b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:08.607428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d918425c-ba9f-49c8-a636-761e5d2e0b3a admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:08.609016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d918425c-ba9f-49c8-a636-761e5d2e0b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:08.609309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d918425c-ba9f-49c8-a636-761e5d2e0b3a admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:08.612825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d918425c-ba9f-49c8-a636-761e5d2e0b3a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:08.620094 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d918425c-ba9f-49c8-a636-761e5d2e0b3a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:09.256490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1123/2243] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:37:08 2022] POST /identity/v3/projects => generated 363 bytes in 685 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:37:09.257720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:09.272487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff5123b-88af-4f8a-a7da-35110535ff42 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:09.354993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f20f7eb-07dc-4743-9887-dba71a051e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:09.356264 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f20f7eb-07dc-4743-9887-dba71a051e90 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:09.356600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f20f7eb-07dc-4743-9887-dba71a051e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:09.356932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f20f7eb-07dc-4743-9887-dba71a051e90 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:09.357780 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f20f7eb-07dc-4743-9887-dba71a051e90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:09.359851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f20f7eb-07dc-4743-9887-dba71a051e90 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:09.479297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1121/2244] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:37:09 2022] POST /identity/v3/users => generated 282 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:37:09.480218 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d918425c-ba9f-49c8-a636-761e5d2e0b3a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:09.495035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d918425c-ba9f-49c8-a636-761e5d2e0b3a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:09.517251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b086318-712d-4f50-b5a4-1f9b0c5b8555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:09.518052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b086318-712d-4f50-b5a4-1f9b0c5b8555 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:09.518224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b086318-712d-4f50-b5a4-1f9b0c5b8555 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:09.519600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b086318-712d-4f50-b5a4-1f9b0c5b8555 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:09.519600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b086318-712d-4f50-b5a4-1f9b0c5b8555 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:09.522231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b086318-712d-4f50-b5a4-1f9b0c5b8555 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:09.537509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1124/2245] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:09 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:37:09.539211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f20f7eb-07dc-4743-9887-dba71a051e90 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:09.564859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f20f7eb-07dc-4743-9887-dba71a051e90 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:09.570976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5321c209-8106-4e5d-8300-b761e156ef72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:09.571872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5321c209-8106-4e5d-8300-b761e156ef72 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:09.572304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5321c209-8106-4e5d-8300-b761e156ef72 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:09.572780 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5321c209-8106-4e5d-8300-b761e156ef72 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:09.573436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5321c209-8106-4e5d-8300-b761e156ef72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:09.575358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5321c209-8106-4e5d-8300-b761e156ef72 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:09.592607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1122/2246] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:09 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:37:09.593235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b086318-712d-4f50-b5a4-1f9b0c5b8555 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:09.619073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b086318-712d-4f50-b5a4-1f9b0c5b8555 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:09.619073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:09.619916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:09.620196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:09.620584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] PATH_INFO: `/v3/projects/8ea4c15b8b634952b049c33db79612bb/users/7cb591d8589944f0b46ee896c26dd0df/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:09.642176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ea4c15b8b634952b049c33db79612bb, user_id=7cb591d8589944f0b46ee896c26dd0df, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:09.645110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:09.826044 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1125/2247] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:37:09 2022] PUT /identity/v3/projects/8ea4c15b8b634952b049c33db79612bb/users/7cb591d8589944f0b46ee896c26dd0df/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:09.836742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f06a0f-862b-4ef5-9f42-685245630d84 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:09.837135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f06a0f-862b-4ef5-9f42-685245630d84 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:09.837425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f06a0f-862b-4ef5-9f42-685245630d84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:09.892182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-31f06a0f-862b-4ef5-9f42-685245630d84 None None] MFA Rules not processed for user `7cb591d8589944f0b46ee896c26dd0df`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:09.893902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-31f06a0f-862b-4ef5-9f42-685245630d84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:09.966094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=4,keystone:DELETE=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:10.000866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1123/2248] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:09 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:10.012870 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:10.018545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:10.104778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fac9003-9ac2-4ef9-8ca4-64639ab4e70f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:10.109369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a0bc092-908b-44b8-99ff-3e31655a1a21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:10.109965 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0bc092-908b-44b8-99ff-3e31655a1a21 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:10.110387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0bc092-908b-44b8-99ff-3e31655a1a21 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:10.110580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0bc092-908b-44b8-99ff-3e31655a1a21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:10.120551 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0bc092-908b-44b8-99ff-3e31655a1a21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:10.122016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a0bc092-908b-44b8-99ff-3e31655a1a21 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:10.133730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1126/2249] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:10 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:12.341701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f06a0f-862b-4ef5-9f42-685245630d84 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:12.343571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-31f06a0f-862b-4ef5-9f42-685245630d84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:12.424318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f06a0f-862b-4ef5-9f42-685245630d84 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:12.428992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:12.429632 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:12.429913 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:12.430188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] PATH_INFO: `/v3/users/c807be8622c94fdb9c9e3688ed2e72db` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:12.431400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c807be8622c94fdb9c9e3688ed2e72db)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:12.433358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:12.626051 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c807be8622c94fdb9c9e3688ed2e72db'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:12.626771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c807be8622c94fdb9c9e3688ed2e72db'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:12.640226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c807be8622c94fdb9c9e3688ed2e72db'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:12.647234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c807be8622c94fdb9c9e3688ed2e72db'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:12.647954 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c807be8622c94fdb9c9e3688ed2e72db'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:12.758520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1124/2250] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:37:12 2022] DELETE /identity/v3/users/c807be8622c94fdb9c9e3688ed2e72db => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:12.763753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ccc30b-0c9a-4afd-9491-68c764e20819 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:12.764215 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ccc30b-0c9a-4afd-9491-68c764e20819 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:12.764306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0ccc30b-0c9a-4afd-9491-68c764e20819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:12.782868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a0ccc30b-0c9a-4afd-9491-68c764e20819 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:12.784105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a0ccc30b-0c9a-4afd-9491-68c764e20819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:12.809910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:12.857168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1127/2251] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:37:12.866887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:12.868323 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:12.931695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca795b9-7b58-4a47-bc24-c76396b49d7e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:12.936245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d73085-15b6-455a-80c5-fe6a72859d1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:12.936800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d73085-15b6-455a-80c5-fe6a72859d1b service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:12.937059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d73085-15b6-455a-80c5-fe6a72859d1b service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:12.937333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d73085-15b6-455a-80c5-fe6a72859d1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:12.947681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d73085-15b6-455a-80c5-fe6a72859d1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:12.949799 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d73085-15b6-455a-80c5-fe6a72859d1b service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:13.177522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:13.177522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1125/2252] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:13.560875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ccc30b-0c9a-4afd-9491-68c764e20819 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:13.568343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a0ccc30b-0c9a-4afd-9491-68c764e20819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:13.585289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0ccc30b-0c9a-4afd-9491-68c764e20819 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:13.597238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:13.597238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:13.597238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:13.597238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] PATH_INFO: `/v3/projects/fbe38f948f7942c4937fca763dbc1959` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:13.597710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fbe38f948f7942c4937fca763dbc1959)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:13.597710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:13.750498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fbe38f948f7942c4937fca763dbc1959', 'request_id': 'req-f538afd5-8b55-46e1-a068-435baa74f917'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:13.829525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fbe38f948f7942c4937fca763dbc1959', 'request_id': 'req-f538afd5-8b55-46e1-a068-435baa74f917'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:13.836823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fbe38f948f7942c4937fca763dbc1959', 'request_id': 'req-f538afd5-8b55-46e1-a068-435baa74f917'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:13.850405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] The token cache is being invalidate because project fbe38f948f7942c4937fca763dbc1959 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:37:13.853633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:13.853633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:13.853633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1128/2253] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:37:13 2022] DELETE /identity/v3/projects/fbe38f948f7942c4937fca763dbc1959 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:13.857078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d73085-15b6-455a-80c5-fe6a72859d1b service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:13.866993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d73085-15b6-455a-80c5-fe6a72859d1b service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:13.934437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:13.935091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:13.935374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:13.935805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] PATH_INFO: `/v3/users/0306b9f5d8d440689407fa2ba0987e6f` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:13.936937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] RBAC: Authorizing `identity:delete_user(user_id=0306b9f5d8d440689407fa2ba0987e6f)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:13.938866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:14.133293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0306b9f5d8d440689407fa2ba0987e6f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:14.133969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0306b9f5d8d440689407fa2ba0987e6f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:14.146427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0306b9f5d8d440689407fa2ba0987e6f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:14.153291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0306b9f5d8d440689407fa2ba0987e6f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:14.153970 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-16d7f9c9-e011-4f3a-84d8-ae1ceeae298a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0306b9f5d8d440689407fa2ba0987e6f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:14.239564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1126/2254] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:37:13 2022] DELETE /identity/v3/users/0306b9f5d8d440689407fa2ba0987e6f => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:37:14.721625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:14.723137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:14.737776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f538afd5-8b55-46e1-a068-435baa74f917 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:14.799930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:14.800563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:14.803272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:14.803272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] PATH_INFO: `/v3/projects/85e80f6957e04b2d91358a8456d2d9f1` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:14.803272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] RBAC: Authorizing `identity:delete_project(project_id=85e80f6957e04b2d91358a8456d2d9f1)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:14.804266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:14.945326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '85e80f6957e04b2d91358a8456d2d9f1', 'request_id': 'req-064f174a-2681-4a48-8cdb-7f33d4878061'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:14.949644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:15.016579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '85e80f6957e04b2d91358a8456d2d9f1', 'request_id': 'req-064f174a-2681-4a48-8cdb-7f33d4878061'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:15.023881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '85e80f6957e04b2d91358a8456d2d9f1', 'request_id': 'req-064f174a-2681-4a48-8cdb-7f33d4878061'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:15.037161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] The token cache is being invalidate because project 85e80f6957e04b2d91358a8456d2d9f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:37:15.039876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:15.039876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-064f174a-2681-4a48-8cdb-7f33d4878061 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:15.039876 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1129/2255] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:37:14 2022] DELETE /identity/v3/projects/85e80f6957e04b2d91358a8456d2d9f1 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:15.045201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4f6ba3-8292-43c6-9b16-53183f4d3a86 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.045707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4f6ba3-8292-43c6-9b16-53183f4d3a86 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.045816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4f6ba3-8292-43c6-9b16-53183f4d3a86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.066284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6a4f6ba3-8292-43c6-9b16-53183f4d3a86 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:15.067551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6a4f6ba3-8292-43c6-9b16-53183f4d3a86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:15.139638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1127/2256] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:37:15.147495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6c220-14bf-43af-8354-4949bb273e50 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.147780 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6c220-14bf-43af-8354-4949bb273e50 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.148062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbd6c220-14bf-43af-8354-4949bb273e50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.169046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-cbd6c220-14bf-43af-8354-4949bb273e50 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:15.170244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-cbd6c220-14bf-43af-8354-4949bb273e50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:15.178656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1130/2257] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:15.182749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4f6ba3-8292-43c6-9b16-53183f4d3a86 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:15.192678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4f6ba3-8292-43c6-9b16-53183f4d3a86 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:15.197390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:15.198550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.198851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.199158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.199826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:37:15.200111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:15.201727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:15.213663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1128/2258] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:37:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:37:15.218776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd6c220-14bf-43af-8354-4949bb273e50 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:15.228564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbd6c220-14bf-43af-8354-4949bb273e50 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:15.233869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684e56f1-f9a1-4742-b253-35beedefd72a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:15.234450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684e56f1-f9a1-4742-b253-35beedefd72a admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.234723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684e56f1-f9a1-4742-b253-35beedefd72a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.235022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684e56f1-f9a1-4742-b253-35beedefd72a admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.235650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684e56f1-f9a1-4742-b253-35beedefd72a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:15.237597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684e56f1-f9a1-4742-b253-35beedefd72a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:15.327753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1131/2259] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:37:15 2022] POST /identity/v3/projects => generated 385 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:37:15.332063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:15.343127 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9862b33-e86b-4016-8049-77041bb060e3 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:15.348081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:15.400923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a9de29-ffb8-4596-b85a-1e6ba8c3058f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:15.401539 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9de29-ffb8-4596-b85a-1e6ba8c3058f admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.401818 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9de29-ffb8-4596-b85a-1e6ba8c3058f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.402091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a9de29-ffb8-4596-b85a-1e6ba8c3058f admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.403391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a9de29-ffb8-4596-b85a-1e6ba8c3058f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:15.405417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a9de29-ffb8-4596-b85a-1e6ba8c3058f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:15.515253 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1129/2260] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:37:15 2022] POST /identity/v3/users => generated 293 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:37:15.519312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684e56f1-f9a1-4742-b253-35beedefd72a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:15.530660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684e56f1-f9a1-4742-b253-35beedefd72a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:15.535773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6275f4d7-db2b-4bc8-b6a1-3fee00ae8c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:15.535940 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6275f4d7-db2b-4bc8-b6a1-3fee00ae8c59 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.536191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6275f4d7-db2b-4bc8-b6a1-3fee00ae8c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.536464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6275f4d7-db2b-4bc8-b6a1-3fee00ae8c59 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.537111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6275f4d7-db2b-4bc8-b6a1-3fee00ae8c59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:15.538713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6275f4d7-db2b-4bc8-b6a1-3fee00ae8c59 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:15.548700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1132/2261] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:37:15.552562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a9de29-ffb8-4596-b85a-1e6ba8c3058f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:15.562402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a9de29-ffb8-4596-b85a-1e6ba8c3058f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:15.567236 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d33be22-1136-4a1f-9031-e7fb1e2ae668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:15.567826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33be22-1136-4a1f-9031-e7fb1e2ae668 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.568108 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33be22-1136-4a1f-9031-e7fb1e2ae668 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.568383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d33be22-1136-4a1f-9031-e7fb1e2ae668 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.569116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d33be22-1136-4a1f-9031-e7fb1e2ae668 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:15.570702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d33be22-1136-4a1f-9031-e7fb1e2ae668 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:15.581892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1130/2262] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:15 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:37:15.586257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6275f4d7-db2b-4bc8-b6a1-3fee00ae8c59 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:15.597489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6275f4d7-db2b-4bc8-b6a1-3fee00ae8c59 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:15.602240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1538bed3-44c8-4972-abcf-301589498dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:15.602914 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538bed3-44c8-4972-abcf-301589498dba admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.603235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538bed3-44c8-4972-abcf-301589498dba admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.603510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538bed3-44c8-4972-abcf-301589498dba admin admin] PATH_INFO: `/v3/projects/7cd5ea95151d456381cdafd3783163ad/users/d21f723938b848a9bfa7ce3b72ff229a/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.618983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1538bed3-44c8-4972-abcf-301589498dba admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cd5ea95151d456381cdafd3783163ad, user_id=d21f723938b848a9bfa7ce3b72ff229a, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:15.622286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1538bed3-44c8-4972-abcf-301589498dba admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:15.724825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1133/2263] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:37:15 2022] PUT /identity/v3/projects/7cd5ea95151d456381cdafd3783163ad/users/d21f723938b848a9bfa7ce3b72ff229a/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:15.735279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ee2fe6-907c-4e0d-8b4f-e1e0e1cb5059 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.735565 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ee2fe6-907c-4e0d-8b4f-e1e0e1cb5059 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.735835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ee2fe6-907c-4e0d-8b4f-e1e0e1cb5059 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.765899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-88ee2fe6-907c-4e0d-8b4f-e1e0e1cb5059 None None] MFA Rules not processed for user `d21f723938b848a9bfa7ce3b72ff229a`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:15.767203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-88ee2fe6-907c-4e0d-8b4f-e1e0e1cb5059 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:15.835989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1131/2264] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:15 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:15.840815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1538bed3-44c8-4972-abcf-301589498dba admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:15.903288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1538bed3-44c8-4972-abcf-301589498dba admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:15.908058 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e908ec00-14cf-4738-aae1-22f82db08e20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:15.908665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e908ec00-14cf-4738-aae1-22f82db08e20 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:15.908947 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e908ec00-14cf-4738-aae1-22f82db08e20 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:15.909225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e908ec00-14cf-4738-aae1-22f82db08e20 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:15.909879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e908ec00-14cf-4738-aae1-22f82db08e20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:15.911845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e908ec00-14cf-4738-aae1-22f82db08e20 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:15.998932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1134/2265] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:37:15 2022] POST /identity/v3/projects => generated 387 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:37:16.000232 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ee2fe6-907c-4e0d-8b4f-e1e0e1cb5059 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:16.012401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ee2fe6-907c-4e0d-8b4f-e1e0e1cb5059 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:16.078941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed79bed-1827-46a7-9da8-f493b0bcd7ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:16.087098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed79bed-1827-46a7-9da8-f493b0bcd7ff admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.087098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed79bed-1827-46a7-9da8-f493b0bcd7ff admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.087098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed79bed-1827-46a7-9da8-f493b0bcd7ff admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.087098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed79bed-1827-46a7-9da8-f493b0bcd7ff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:16.087098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed79bed-1827-46a7-9da8-f493b0bcd7ff admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:16.191289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1132/2266] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:37:15 2022] POST /identity/v3/users => generated 300 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:37:16.192849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e908ec00-14cf-4738-aae1-22f82db08e20 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:16.211431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e908ec00-14cf-4738-aae1-22f82db08e20 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:16.228427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908fde2b-4436-42bc-8a63-96acab23ac5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:16.229394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908fde2b-4436-42bc-8a63-96acab23ac5f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.229954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908fde2b-4436-42bc-8a63-96acab23ac5f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.231081 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908fde2b-4436-42bc-8a63-96acab23ac5f admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.231983 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908fde2b-4436-42bc-8a63-96acab23ac5f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:16.235779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908fde2b-4436-42bc-8a63-96acab23ac5f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:16.257805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1135/2267] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:37:16.262825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed79bed-1827-46a7-9da8-f493b0bcd7ff admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:16.270454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:16.274664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed79bed-1827-46a7-9da8-f493b0bcd7ff admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:16.281036 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0da1919-e4d8-420a-8635-11fbc15aac2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:16.281713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da1919-e4d8-420a-8635-11fbc15aac2b admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.282156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da1919-e4d8-420a-8635-11fbc15aac2b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.282431 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0da1919-e4d8-420a-8635-11fbc15aac2b admin admin] PATH_INFO: `/v3/projects/eafaee5cc42742c2bc21abbdd0a35ae4/users/1d498b98e5c84cdc8051d14db3bc7e93/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.300110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0da1919-e4d8-420a-8635-11fbc15aac2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=eafaee5cc42742c2bc21abbdd0a35ae4, user_id=1d498b98e5c84cdc8051d14db3bc7e93, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:16.302922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0da1919-e4d8-420a-8635-11fbc15aac2b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:16.394060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1133/2268] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:37:16 2022] PUT /identity/v3/projects/eafaee5cc42742c2bc21abbdd0a35ae4/users/1d498b98e5c84cdc8051d14db3bc7e93/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:16.399001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908fde2b-4436-42bc-8a63-96acab23ac5f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:16.411477 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908fde2b-4436-42bc-8a63-96acab23ac5f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:16.468393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:16.483743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecffd71d-0524-4bd0-bc0c-32cbdbf4af75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:16.483743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecffd71d-0524-4bd0-bc0c-32cbdbf4af75 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.483743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecffd71d-0524-4bd0-bc0c-32cbdbf4af75 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.483743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecffd71d-0524-4bd0-bc0c-32cbdbf4af75 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.484441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecffd71d-0524-4bd0-bc0c-32cbdbf4af75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:16.486182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecffd71d-0524-4bd0-bc0c-32cbdbf4af75 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:16.500743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1136/2269] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:16 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:37:16.506311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0da1919-e4d8-420a-8635-11fbc15aac2b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:16.512762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0da1919-e4d8-420a-8635-11fbc15aac2b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:16.517468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336f121a-bb80-4036-944f-05e6f108d80e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:16.518254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f121a-bb80-4036-944f-05e6f108d80e admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.518537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f121a-bb80-4036-944f-05e6f108d80e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.518813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336f121a-bb80-4036-944f-05e6f108d80e admin admin] PATH_INFO: `/v3/projects/eafaee5cc42742c2bc21abbdd0a35ae4/users/1d498b98e5c84cdc8051d14db3bc7e93/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.520798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f121a-bb80-4036-944f-05e6f108d80e admin admin] RBAC: Authorizing `identity:create_grant(project_id=eafaee5cc42742c2bc21abbdd0a35ae4, user_id=1d498b98e5c84cdc8051d14db3bc7e93, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:16.523493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336f121a-bb80-4036-944f-05e6f108d80e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:16.537525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1134/2270] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:37:16 2022] PUT /identity/v3/projects/eafaee5cc42742c2bc21abbdd0a35ae4/users/1d498b98e5c84cdc8051d14db3bc7e93/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:16.549028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458a4a0-5c68-46e3-a046-a6675b626aa2 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.549308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458a4a0-5c68-46e3-a046-a6675b626aa2 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.549578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458a4a0-5c68-46e3-a046-a6675b626aa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.583804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-3458a4a0-5c68-46e3-a046-a6675b626aa2 None None] MFA Rules not processed for user `1d498b98e5c84cdc8051d14db3bc7e93`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:16.585034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3458a4a0-5c68-46e3-a046-a6675b626aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:16.586592 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654ead2e-5be9-4f36-8c84-95d5e6e92e44 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.586887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654ead2e-5be9-4f36-8c84-95d5e6e92e44 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.587205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654ead2e-5be9-4f36-8c84-95d5e6e92e44 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.588757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1135/2271] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:37:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 19 22:37:16.595040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc364c-35de-401a-9f80-e93777938032 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.595329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc364c-35de-401a-9f80-e93777938032 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.595610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dc364c-35de-401a-9f80-e93777938032 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.641965 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-83dc364c-35de-401a-9f80-e93777938032 None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:16.643580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-83dc364c-35de-401a-9f80-e93777938032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:16.685928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1137/2272] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:16 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:37:16.686577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3458a4a0-5c68-46e3-a046-a6675b626aa2 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:16.687654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3458a4a0-5c68-46e3-a046-a6675b626aa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:16.727462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1136/2273] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:37:16 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:37:16.740523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc364c-35de-401a-9f80-e93777938032 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:16.742055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-83dc364c-35de-401a-9f80-e93777938032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:16.759625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3458a4a0-5c68-46e3-a046-a6675b626aa2 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:16.764444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5f0455-780e-4f9e-b20e-999cb2164fee service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:16.765160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5f0455-780e-4f9e-b20e-999cb2164fee service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.765552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5f0455-780e-4f9e-b20e-999cb2164fee service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.765892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5f0455-780e-4f9e-b20e-999cb2164fee service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.776933 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5f0455-780e-4f9e-b20e-999cb2164fee service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:16.779330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5f0455-780e-4f9e-b20e-999cb2164fee service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:16.799093 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1138/2274] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:16 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:37:16.811619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dc364c-35de-401a-9f80-e93777938032 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:16.821613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5f0455-780e-4f9e-b20e-999cb2164fee service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:16.823341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e874cc1-e54c-43b7-8b52-c1755721965c service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:16.829612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e874cc1-e54c-43b7-8b52-c1755721965c service glance] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.829612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e874cc1-e54c-43b7-8b52-c1755721965c service glance] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.829612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e874cc1-e54c-43b7-8b52-c1755721965c service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.836067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e874cc1-e54c-43b7-8b52-c1755721965c service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:16.837541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e874cc1-e54c-43b7-8b52-c1755721965c service glance] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:16.850409 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1137/2275] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:16 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:16.884816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5f0455-780e-4f9e-b20e-999cb2164fee service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:16.890261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:16.891390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:16.891390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:16.891685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] PATH_INFO: `/v3/users/d21f723938b848a9bfa7ce3b72ff229a` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:16.892974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d21f723938b848a9bfa7ce3b72ff229a)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:16.894937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:17.091957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c132e7d3-c910-4308-8b95-c2688059ab46 None None] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.092491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c132e7d3-c910-4308-8b95-c2688059ab46 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.092491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c132e7d3-c910-4308-8b95-c2688059ab46 None None] PATH_INFO: `/` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.093827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1138/2276] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:37:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:37:17.098215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797d5ee3-8f51-4435-9f51-715c08f69223 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.101879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797d5ee3-8f51-4435-9f51-715c08f69223 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.101879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797d5ee3-8f51-4435-9f51-715c08f69223 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.102243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd21f723938b848a9bfa7ce3b72ff229a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:17.137942 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-797d5ee3-8f51-4435-9f51-715c08f69223 None None] MFA Rules not processed for user `2f705dc9d2e541f1821dda4b5c7b7709`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:17.139176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-797d5ee3-8f51-4435-9f51-715c08f69223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:17.164819 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd21f723938b848a9bfa7ce3b72ff229a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:17.169665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:17.177665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd21f723938b848a9bfa7ce3b72ff229a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:17.185727 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd21f723938b848a9bfa7ce3b72ff229a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:17.186588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd21f723938b848a9bfa7ce3b72ff229a'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:17.190922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1139/2277] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:37:17 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:37:17.192797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1139/2278] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:37:16 2022] DELETE /identity/v3/users/d21f723938b848a9bfa7ce3b72ff229a => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:17.198608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174c9450-2979-444b-b880-47ec62bc20ab None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.198608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174c9450-2979-444b-b880-47ec62bc20ab None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.198608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-174c9450-2979-444b-b880-47ec62bc20ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.203674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:17.205177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:17.217824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-174c9450-2979-444b-b880-47ec62bc20ab None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:17.219329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-174c9450-2979-444b-b880-47ec62bc20ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:17.251231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:17.265118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7caf3950-4ed7-4dc6-a7db-781e0e9dec58 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:17.268834 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4493f66e-088e-491d-8bec-863368d2489b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:17.269431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4493f66e-088e-491d-8bec-863368d2489b service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.269730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4493f66e-088e-491d-8bec-863368d2489b service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.269999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4493f66e-088e-491d-8bec-863368d2489b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.277403 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1140/2279] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:17.277556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4493f66e-088e-491d-8bec-863368d2489b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:17.278888 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4493f66e-088e-491d-8bec-863368d2489b service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:17.287057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174c9450-2979-444b-b880-47ec62bc20ab None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:17.287419 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1140/2280] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:17.288418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-174c9450-2979-444b-b880-47ec62bc20ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:17.310107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-174c9450-2979-444b-b880-47ec62bc20ab None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:17.314607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5bc5cd-c093-4f4d-addd-43a0925db3a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:17.315501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5bc5cd-c093-4f4d-addd-43a0925db3a8 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.315501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5bc5cd-c093-4f4d-addd-43a0925db3a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.315955 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5bc5cd-c093-4f4d-addd-43a0925db3a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.331321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5bc5cd-c093-4f4d-addd-43a0925db3a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:17.332773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5bc5cd-c093-4f4d-addd-43a0925db3a8 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:17.343473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1141/2281] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:37:17.382666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dae8123-eb0a-403b-9bad-cff927b24214 None None] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.382975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dae8123-eb0a-403b-9bad-cff927b24214 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.383305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dae8123-eb0a-403b-9bad-cff927b24214 None None] PATH_INFO: `/` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.385403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1141/2282] 10.222.0.20 () {54 vars in 921 bytes} [Wed Oct 19 22:37:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 19 22:37:17.391652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f5afaf-9538-41ff-a97a-799d051193bd None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.391995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f5afaf-9538-41ff-a97a-799d051193bd None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.392215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f5afaf-9538-41ff-a97a-799d051193bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.414700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-69f5afaf-9538-41ff-a97a-799d051193bd None None] MFA Rules not processed for user `35fc6d90245c4939bc7171201eb883bd`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:17.417336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-69f5afaf-9538-41ff-a97a-799d051193bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:17.491442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1142/2283] 10.222.0.20 () {58 vars in 1018 bytes} [Wed Oct 19 22:37:17 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:17.502235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dae8123-eb0a-403b-9bad-cff927b24214 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:17.514635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dae8123-eb0a-403b-9bad-cff927b24214 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:17.518325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c4661-8898-41b3-b308-9d4b12944bb4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:17.518947 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c4661-8898-41b3-b308-9d4b12944bb4 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.525409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c4661-8898-41b3-b308-9d4b12944bb4 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.525409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733c4661-8898-41b3-b308-9d4b12944bb4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.529761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733c4661-8898-41b3-b308-9d4b12944bb4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:17.531230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733c4661-8898-41b3-b308-9d4b12944bb4 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:17.542013 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1142/2284] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:17 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:17.893616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f5afaf-9538-41ff-a97a-799d051193bd None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:17.895580 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-69f5afaf-9538-41ff-a97a-799d051193bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:17.909938 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f5afaf-9538-41ff-a97a-799d051193bd None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:17.915647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:17.916298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:17.916554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:17.916822 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] PATH_INFO: `/v3/projects/7cd5ea95151d456381cdafd3783163ad` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:17.918895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cd5ea95151d456381cdafd3783163ad)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:17.923772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:18.059709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cd5ea95151d456381cdafd3783163ad', 'request_id': 'req-151575ac-ce54-451a-b24e-dff738598704'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.067819 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cd5ea95151d456381cdafd3783163ad', 'request_id': 'req-151575ac-ce54-451a-b24e-dff738598704'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.074360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:18.075881 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cd5ea95151d456381cdafd3783163ad', 'request_id': 'req-151575ac-ce54-451a-b24e-dff738598704'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.204789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] The token cache is being invalidate because project 7cd5ea95151d456381cdafd3783163ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:37:18.205462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.205981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.211007 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1143/2285] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:37:17 2022] DELETE /identity/v3/projects/7cd5ea95151d456381cdafd3783163ad => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:18.212339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c4661-8898-41b3-b308-9d4b12944bb4 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:18.225359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733c4661-8898-41b3-b308-9d4b12944bb4 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:18.282504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:18.283161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:18.283439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:18.283708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] PATH_INFO: `/v3/users/1d498b98e5c84cdc8051d14db3bc7e93` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:18.284866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d498b98e5c84cdc8051d14db3bc7e93)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:18.286786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:18.367705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:18.477859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d498b98e5c84cdc8051d14db3bc7e93'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.547137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d498b98e5c84cdc8051d14db3bc7e93'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.559609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d498b98e5c84cdc8051d14db3bc7e93'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.566436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d498b98e5c84cdc8051d14db3bc7e93'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.567153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-74202023-7a96-4c16-9287-cd85d8bc3108 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d498b98e5c84cdc8051d14db3bc7e93'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:18.573160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1143/2286] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:37:18 2022] DELETE /identity/v3/users/1d498b98e5c84cdc8051d14db3bc7e93 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:37:19.003907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:19.005358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:19.019422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151575ac-ce54-451a-b24e-dff738598704 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:19.116366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:19.117022 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.117302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.117572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] PATH_INFO: `/v3/projects/eafaee5cc42742c2bc21abbdd0a35ae4` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.118761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] RBAC: Authorizing `identity:delete_project(project_id=eafaee5cc42742c2bc21abbdd0a35ae4)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:19.120750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:19.225550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eafaee5cc42742c2bc21abbdd0a35ae4', 'request_id': 'req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:19.233097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eafaee5cc42742c2bc21abbdd0a35ae4', 'request_id': 'req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:19.239582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eafaee5cc42742c2bc21abbdd0a35ae4', 'request_id': 'req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:19.352986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] The token cache is being invalidate because project eafaee5cc42742c2bc21abbdd0a35ae4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:37:19.355246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:19.355246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-fb3ace3c-fe12-47f0-ac48-6816eb99cf3a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:19.355963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1144/2287] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:37:19 2022] DELETE /identity/v3/projects/eafaee5cc42742c2bc21abbdd0a35ae4 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:19.361105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041eca5f-1735-498c-a41d-0ec44794523b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.361593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041eca5f-1735-498c-a41d-0ec44794523b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.361692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-041eca5f-1735-498c-a41d-0ec44794523b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.381852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-041eca5f-1735-498c-a41d-0ec44794523b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:19.383110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-041eca5f-1735-498c-a41d-0ec44794523b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:19.446470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1144/2288] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:19.454346 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936cafe-3f3c-4fee-b00d-297352e20d54 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.454628 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936cafe-3f3c-4fee-b00d-297352e20d54 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.455257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a936cafe-3f3c-4fee-b00d-297352e20d54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.474879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a936cafe-3f3c-4fee-b00d-297352e20d54 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:19.476106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a936cafe-3f3c-4fee-b00d-297352e20d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:19.484598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1145/2289] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:19.488672 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041eca5f-1735-498c-a41d-0ec44794523b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:19.498458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-041eca5f-1735-498c-a41d-0ec44794523b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:19.503149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:19.507011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.507011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.507011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.507011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:37:19.507011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:19.507011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:19.521713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1145/2290] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:37:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:37:19.522765 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a936cafe-3f3c-4fee-b00d-297352e20d54 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:19.533699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a936cafe-3f3c-4fee-b00d-297352e20d54 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:19.538447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d9a357-e392-47d4-b09e-3aa6669f95c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:19.539072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d9a357-e392-47d4-b09e-3aa6669f95c4 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.539353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d9a357-e392-47d4-b09e-3aa6669f95c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.539627 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d9a357-e392-47d4-b09e-3aa6669f95c4 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.540254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d9a357-e392-47d4-b09e-3aa6669f95c4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:19.542139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d9a357-e392-47d4-b09e-3aa6669f95c4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:19.639027 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1146/2291] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:37:19 2022] POST /identity/v3/projects => generated 377 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:37:19.640307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:19.650640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a684109a-927d-4e5a-96ab-f9ef755258f9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:19.709092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5a2962-2a91-4063-a5ef-9c72af619bb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:19.709699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a2962-2a91-4063-a5ef-9c72af619bb3 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.709975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a2962-2a91-4063-a5ef-9c72af619bb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.710257 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5a2962-2a91-4063-a5ef-9c72af619bb3 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.710874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5a2962-2a91-4063-a5ef-9c72af619bb3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:19.712985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5a2962-2a91-4063-a5ef-9c72af619bb3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:19.807160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1146/2292] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:37:19 2022] POST /identity/v3/users => generated 289 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:37:19.811388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d9a357-e392-47d4-b09e-3aa6669f95c4 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:19.826501 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d9a357-e392-47d4-b09e-3aa6669f95c4 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:19.835072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e785bf37-599f-47f8-9519-8856970bc089 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:19.835740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e785bf37-599f-47f8-9519-8856970bc089 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.836019 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e785bf37-599f-47f8-9519-8856970bc089 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.836291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e785bf37-599f-47f8-9519-8856970bc089 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.836937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e785bf37-599f-47f8-9519-8856970bc089 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:19.839238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e785bf37-599f-47f8-9519-8856970bc089 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:19.850978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1147/2293] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:37:19.855053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5a2962-2a91-4063-a5ef-9c72af619bb3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:19.865201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5a2962-2a91-4063-a5ef-9c72af619bb3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:19.870762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa5fda71-104f-4d7b-a9fe-152d6d77b915 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:19.871431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5fda71-104f-4d7b-a9fe-152d6d77b915 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.871717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5fda71-104f-4d7b-a9fe-152d6d77b915 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.871992 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa5fda71-104f-4d7b-a9fe-152d6d77b915 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.872746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5fda71-104f-4d7b-a9fe-152d6d77b915 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:19.874366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa5fda71-104f-4d7b-a9fe-152d6d77b915 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:19.885212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1147/2294] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:37:19.889915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e785bf37-599f-47f8-9519-8856970bc089 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:19.900015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e785bf37-599f-47f8-9519-8856970bc089 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:19.905307 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:19.905947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.906235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.906506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] PATH_INFO: `/v3/projects/09f5595e60574e06ae13840d24e75f2b/users/652bcd71d0214ba8b877be84c5afd812/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:19.921554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] RBAC: Authorizing `identity:create_grant(project_id=09f5595e60574e06ae13840d24e75f2b, user_id=652bcd71d0214ba8b877be84c5afd812, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:19.924295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:19.982568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1148/2295] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:37:19 2022] PUT /identity/v3/projects/09f5595e60574e06ae13840d24e75f2b/users/652bcd71d0214ba8b877be84c5afd812/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:19.994186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0401efe4-548f-47d0-a44f-0589ddc0a537 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:19.994520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0401efe4-548f-47d0-a44f-0589ddc0a537 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:19.994795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0401efe4-548f-47d0-a44f-0589ddc0a537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:20.016322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:20.026871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0401efe4-548f-47d0-a44f-0589ddc0a537 None None] MFA Rules not processed for user `652bcd71d0214ba8b877be84c5afd812`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:20.026871 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0401efe4-548f-47d0-a44f-0589ddc0a537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:20.089646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1148/2296] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:19 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:20.101117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:20.102908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:20.118042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:20.172070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574d8c27-dd93-483e-bb14-e574dbb4f72e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:20.176224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:20.177277 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:20.180137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:20.180137 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:20.186644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:20.188120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:20.197914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1149/2297] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:20 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:23.749700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4b6d3-0498-4576-8b1d-255a8b175187 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:23.750262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4b6d3-0498-4576-8b1d-255a8b175187 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:23.750407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b4b6d3-0498-4576-8b1d-255a8b175187 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:23.771138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-55b4b6d3-0498-4576-8b1d-255a8b175187 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:23.772382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-55b4b6d3-0498-4576-8b1d-255a8b175187 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:23.837720 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1149/2298] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:37:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:23.851953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:23.853474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:23.925995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec7db201-2466-40b8-80d2-b8e9e735adf4 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:23.931026 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d180aa-4807-4460-9532-3c5bae628db4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:23.931635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d180aa-4807-4460-9532-3c5bae628db4 service swift] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:23.931911 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d180aa-4807-4460-9532-3c5bae628db4 service swift] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:23.932180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d180aa-4807-4460-9532-3c5bae628db4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:23.941310 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d180aa-4807-4460-9532-3c5bae628db4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:23.942813 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d180aa-4807-4460-9532-3c5bae628db4 service swift] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:23.952071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1150/2299] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:37:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:37:33.829846 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=52 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:33.947893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=52 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:42.106526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea213a36-48ce-4671-bd21-091c88293a66 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:42.109647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea213a36-48ce-4671-bd21-091c88293a66 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:42.109647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea213a36-48ce-4671-bd21-091c88293a66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:42.128474 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ea213a36-48ce-4671-bd21-091c88293a66 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:42.128705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ea213a36-48ce-4671-bd21-091c88293a66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:42.136409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1150/2300] 10.222.0.20 () {58 vars in 967 bytes} [Wed Oct 19 22:37:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:42.148833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d180aa-4807-4460-9532-3c5bae628db4 service swift] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:42.162468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d180aa-4807-4460-9532-3c5bae628db4 service swift] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:42.170258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1202d66-7c73-4650-a6df-22305750ad2f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:42.171091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1202d66-7c73-4650-a6df-22305750ad2f service swift] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:42.171857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1202d66-7c73-4650-a6df-22305750ad2f service swift] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:42.172206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1202d66-7c73-4650-a6df-22305750ad2f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:42.182221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1202d66-7c73-4650-a6df-22305750ad2f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:42.183944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1202d66-7c73-4650-a6df-22305750ad2f service swift] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:42.196071 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1151/2301] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:37:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:37:52.122354 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:52.192325 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:53.700456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea213a36-48ce-4671-bd21-091c88293a66 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:53.774342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea213a36-48ce-4671-bd21-091c88293a66 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:53.779208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:53.779960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:53.780522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:53.780522 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] PATH_INFO: `/v3/users/652bcd71d0214ba8b877be84c5afd812` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:53.781669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=652bcd71d0214ba8b877be84c5afd812)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:53.787253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:53.932417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '652bcd71d0214ba8b877be84c5afd812'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:53.998310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '652bcd71d0214ba8b877be84c5afd812'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.011186 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '652bcd71d0214ba8b877be84c5afd812'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.019641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '652bcd71d0214ba8b877be84c5afd812'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.020349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '652bcd71d0214ba8b877be84c5afd812'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.026566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1151/2302] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:37:53 2022] DELETE /identity/v3/users/652bcd71d0214ba8b877be84c5afd812 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:54.031744 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2518a-8ab0-467f-9163-aa0032c8e5da None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:54.032291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2518a-8ab0-467f-9163-aa0032c8e5da None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:54.032441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d2518a-8ab0-467f-9163-aa0032c8e5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:54.055344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e2d2518a-8ab0-467f-9163-aa0032c8e5da None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:54.056568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e2d2518a-8ab0-467f-9163-aa0032c8e5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:54.122482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1152/2303] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:54.132394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:54.133812 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:54.190855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4d8d15-c069-424e-a7fd-a6da70a9ac3d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:54.194397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033ed9d5-7041-4b53-add3-36a0686802da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:54.195032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ed9d5-7041-4b53-add3-36a0686802da service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:54.195309 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ed9d5-7041-4b53-add3-36a0686802da service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:54.195578 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033ed9d5-7041-4b53-add3-36a0686802da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:54.206637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033ed9d5-7041-4b53-add3-36a0686802da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:54.209382 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033ed9d5-7041-4b53-add3-36a0686802da service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:54.220131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1152/2304] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:54.599335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2518a-8ab0-467f-9163-aa0032c8e5da None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:54.600774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e2d2518a-8ab0-467f-9163-aa0032c8e5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:54.613124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d2518a-8ab0-467f-9163-aa0032c8e5da None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:54.617692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:54.618311 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:54.618589 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:54.618863 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] PATH_INFO: `/v3/projects/09f5595e60574e06ae13840d24e75f2b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:54.620382 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09f5595e60574e06ae13840d24e75f2b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:54.622740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:54.752674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09f5595e60574e06ae13840d24e75f2b', 'request_id': 'req-805e10d5-94b8-4c89-bf35-22b860e04fd0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.758827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09f5595e60574e06ae13840d24e75f2b', 'request_id': 'req-805e10d5-94b8-4c89-bf35-22b860e04fd0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.764336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09f5595e60574e06ae13840d24e75f2b', 'request_id': 'req-805e10d5-94b8-4c89-bf35-22b860e04fd0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.850577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] The token cache is being invalidate because project 09f5595e60574e06ae13840d24e75f2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:37:54.851470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.852612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-805e10d5-94b8-4c89-bf35-22b860e04fd0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:37:54.855465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1153/2305] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:37:54 2022] DELETE /identity/v3/projects/09f5595e60574e06ae13840d24e75f2b => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:54.864752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd4eda-07f7-4160-ac6a-be28d4b93f1c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:54.865217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd4eda-07f7-4160-ac6a-be28d4b93f1c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:54.865305 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cd4eda-07f7-4160-ac6a-be28d4b93f1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:54.884976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-38cd4eda-07f7-4160-ac6a-be28d4b93f1c None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:54.886170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-38cd4eda-07f7-4160-ac6a-be28d4b93f1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:54.902451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:54.944441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1153/2306] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:54.952223 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9feb7b-ebd2-4b01-9a7d-0d032d8856d8 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:54.952511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9feb7b-ebd2-4b01-9a7d-0d032d8856d8 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:54.952789 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9feb7b-ebd2-4b01-9a7d-0d032d8856d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:54.973163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6c9feb7b-ebd2-4b01-9a7d-0d032d8856d8 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:54.974416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6c9feb7b-ebd2-4b01-9a7d-0d032d8856d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:54.982281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1154/2307] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:54.987163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cd4eda-07f7-4160-ac6a-be28d4b93f1c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:54.997005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cd4eda-07f7-4160-ac6a-be28d4b93f1c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:55.001762 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:55.002361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:55.002637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:55.002976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:55.003602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:37:55.003907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:55.005501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:55.017107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1154/2308] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:37:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:37:55.021168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9feb7b-ebd2-4b01-9a7d-0d032d8856d8 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:55.030727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9feb7b-ebd2-4b01-9a7d-0d032d8856d8 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:55.035355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb973dd6-7419-4c5a-b2f6-09ed1c017a3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:55.035926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb973dd6-7419-4c5a-b2f6-09ed1c017a3f admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:55.036204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb973dd6-7419-4c5a-b2f6-09ed1c017a3f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:55.036474 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb973dd6-7419-4c5a-b2f6-09ed1c017a3f admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:55.037129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb973dd6-7419-4c5a-b2f6-09ed1c017a3f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:55.039123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb973dd6-7419-4c5a-b2f6-09ed1c017a3f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:55.114401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1155/2309] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:37:55 2022] POST /identity/v3/projects => generated 375 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:37:55.118493 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:55.128629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fed3a17-f7c2-4741-a293-9c20402c6bed admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:55.183552 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248d4ba-d02b-4004-98ad-06d3f99a5252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:55.184146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248d4ba-d02b-4004-98ad-06d3f99a5252 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:55.184424 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248d4ba-d02b-4004-98ad-06d3f99a5252 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:55.184698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d248d4ba-d02b-4004-98ad-06d3f99a5252 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:55.185336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d248d4ba-d02b-4004-98ad-06d3f99a5252 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:55.187301 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d248d4ba-d02b-4004-98ad-06d3f99a5252 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:55.278296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1155/2310] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:37:55 2022] POST /identity/v3/users => generated 288 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:37:55.282623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb973dd6-7419-4c5a-b2f6-09ed1c017a3f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:55.293054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb973dd6-7419-4c5a-b2f6-09ed1c017a3f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:55.298229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae6bad-a88f-4953-8e9b-3f48fa82a992 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:55.298816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae6bad-a88f-4953-8e9b-3f48fa82a992 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:55.299130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae6bad-a88f-4953-8e9b-3f48fa82a992 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:55.299399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaae6bad-a88f-4953-8e9b-3f48fa82a992 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:55.300034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae6bad-a88f-4953-8e9b-3f48fa82a992 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:55.301609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaae6bad-a88f-4953-8e9b-3f48fa82a992 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:55.311355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1156/2311] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:37:55.315108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248d4ba-d02b-4004-98ad-06d3f99a5252 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:55.324978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d248d4ba-d02b-4004-98ad-06d3f99a5252 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:55.330102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cfb00d-9425-42ed-ad71-5e40fec729cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:55.330686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cfb00d-9425-42ed-ad71-5e40fec729cb admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:55.330996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cfb00d-9425-42ed-ad71-5e40fec729cb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:55.331280 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cfb00d-9425-42ed-ad71-5e40fec729cb admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:55.331957 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cfb00d-9425-42ed-ad71-5e40fec729cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:55.333554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cfb00d-9425-42ed-ad71-5e40fec729cb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:55.345921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1156/2312] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:37:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:37:55.349775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae6bad-a88f-4953-8e9b-3f48fa82a992 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:55.359668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaae6bad-a88f-4953-8e9b-3f48fa82a992 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:55.364170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:55.364798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:55.365074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:55.365342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] PATH_INFO: `/v3/projects/6e4ff5b1b2104c2b8432c3bd7e749e5a/users/bb5498b4fb7e4b54ae07bb7659a46ecd/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:55.380870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e4ff5b1b2104c2b8432c3bd7e749e5a, user_id=bb5498b4fb7e4b54ae07bb7659a46ecd, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:55.383622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:55.455557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1157/2313] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:37:55 2022] PUT /identity/v3/projects/6e4ff5b1b2104c2b8432c3bd7e749e5a/users/bb5498b4fb7e4b54ae07bb7659a46ecd/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:37:55.466100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c178e4-236f-43e8-b364-d5ca87c8f4a2 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:55.466462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c178e4-236f-43e8-b364-d5ca87c8f4a2 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:55.466747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c178e4-236f-43e8-b364-d5ca87c8f4a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:55.495053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-b8c178e4-236f-43e8-b364-d5ca87c8f4a2 None None] MFA Rules not processed for user `bb5498b4fb7e4b54ae07bb7659a46ecd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:55.496262 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b8c178e4-236f-43e8-b364-d5ca87c8f4a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:55.558726 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1157/2314] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:55 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:37:55.569274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:37:55.570975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:55.593682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:37:55.627287 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccdf45e-c306-4fe6-8894-d6e80bfd7429 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:37:55.630985 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:37:55.639668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:55.639668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:55.639668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:55.641552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:37:55.643013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:37:55.651541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1158/2315] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:37:55 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:37:57.356676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdad1b-4d1b-4913-af7c-b5345dcc5bbd None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:37:57.357171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdad1b-4d1b-4913-af7c-b5345dcc5bbd None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:37:57.357263 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fdad1b-4d1b-4913-af7c-b5345dcc5bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:37:57.376573 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-d5fdad1b-4d1b-4913-af7c-b5345dcc5bbd None None] MFA Rules not processed for user `bb5498b4fb7e4b54ae07bb7659a46ecd`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:37:57.377765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d5fdad1b-4d1b-4913-af7c-b5345dcc5bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:37:57.384960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1158/2316] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:37:57 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:04.996645 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:04.998211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:05.066030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d910f8-96cc-4ecb-904f-a2f98fc7da52 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:05.069514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:05.070164 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:05.070458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:05.070730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] PATH_INFO: `/v3/users/7cb591d8589944f0b46ee896c26dd0df` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:05.071891 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7cb591d8589944f0b46ee896c26dd0df)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:05.073814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:05.228337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7cb591d8589944f0b46ee896c26dd0df'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:05.228996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7cb591d8589944f0b46ee896c26dd0df'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:05.241237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7cb591d8589944f0b46ee896c26dd0df'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:05.248230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7cb591d8589944f0b46ee896c26dd0df'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:05.248864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7cb591d8589944f0b46ee896c26dd0df'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:05.329082 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1159/2317] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:38:04 2022] DELETE /identity/v3/users/7cb591d8589944f0b46ee896c26dd0df => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:05.333548 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1250ab06-a725-45ac-8dc4-70ad57b88814 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:05.334054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1250ab06-a725-45ac-8dc4-70ad57b88814 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:05.334151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1250ab06-a725-45ac-8dc4-70ad57b88814 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:05.353794 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1250ab06-a725-45ac-8dc4-70ad57b88814 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:05.355003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1250ab06-a725-45ac-8dc4-70ad57b88814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:05.359961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:05.415253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1159/2318] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:05.424824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:05.426224 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:05.473149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:05.495198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b51e90a-a37d-4b7e-8d5b-0d2f545ceaf1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:05.498756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff73df02-b29d-4595-9397-ee496b5a0ac7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:05.499969 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff73df02-b29d-4595-9397-ee496b5a0ac7 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:05.500428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff73df02-b29d-4595-9397-ee496b5a0ac7 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:05.500931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff73df02-b29d-4595-9397-ee496b5a0ac7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:05.510239 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff73df02-b29d-4595-9397-ee496b5a0ac7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:05.511687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff73df02-b29d-4595-9397-ee496b5a0ac7 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:05.521527 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1160/2319] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:05.862673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1250ab06-a725-45ac-8dc4-70ad57b88814 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:05.865149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1250ab06-a725-45ac-8dc4-70ad57b88814 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:05.879354 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1250ab06-a725-45ac-8dc4-70ad57b88814 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:05.884278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:05.884923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:05.885206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:05.885480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] PATH_INFO: `/v3/projects/8ea4c15b8b634952b049c33db79612bb` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:05.886630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ea4c15b8b634952b049c33db79612bb)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:05.888803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:06.008746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ea4c15b8b634952b049c33db79612bb', 'request_id': 'req-f501515f-311b-42c5-8038-4185c853d71c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:06.081946 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ea4c15b8b634952b049c33db79612bb', 'request_id': 'req-f501515f-311b-42c5-8038-4185c853d71c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:06.088287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ea4c15b8b634952b049c33db79612bb', 'request_id': 'req-f501515f-311b-42c5-8038-4185c853d71c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:06.100029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] The token cache is being invalidate because project 8ea4c15b8b634952b049c33db79612bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:38:06.100320 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:06.100991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:06.102709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1160/2320] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:38:05 2022] DELETE /identity/v3/projects/8ea4c15b8b634952b049c33db79612bb => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:15.105401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dbdd71-b1d5-4df2-b6ba-5169cb7119c2 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:15.105862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dbdd71-b1d5-4df2-b6ba-5169cb7119c2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:15.105961 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5dbdd71-b1d5-4df2-b6ba-5169cb7119c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:15.125157 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b5dbdd71-b1d5-4df2-b6ba-5169cb7119c2 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:15.126363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b5dbdd71-b1d5-4df2-b6ba-5169cb7119c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:15.180748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1161/2321] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:38:15 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:15.202827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:15.204240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:15.271633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f501515f-311b-42c5-8038-4185c853d71c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:15.276572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36d9f1b-0500-46b8-8c9f-35c4b0f4dbd6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:15.277166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36d9f1b-0500-46b8-8c9f-35c4b0f4dbd6 service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:15.277442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36d9f1b-0500-46b8-8c9f-35c4b0f4dbd6 service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:15.277737 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36d9f1b-0500-46b8-8c9f-35c4b0f4dbd6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:15.285619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36d9f1b-0500-46b8-8c9f-35c4b0f4dbd6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:15.287253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36d9f1b-0500-46b8-8c9f-35c4b0f4dbd6 service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:15.295371 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1161/2322] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:38:15 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:38:16.209706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962d565-80b0-4935-8723-64aa1bdae58a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:16.210169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962d565-80b0-4935-8723-64aa1bdae58a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:16.210278 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7962d565-80b0-4935-8723-64aa1bdae58a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:16.229338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-7962d565-80b0-4935-8723-64aa1bdae58a None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:16.230516 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7962d565-80b0-4935-8723-64aa1bdae58a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:16.237692 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1162/2323] 10.222.0.20 () {58 vars in 967 bytes} [Wed Oct 19 22:38:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:16.250109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36d9f1b-0500-46b8-8c9f-35c4b0f4dbd6 service swift] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:16.260719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36d9f1b-0500-46b8-8c9f-35c4b0f4dbd6 service swift] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:16.266648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc87349-699b-4b7c-ba3f-a7eb5ebec838 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:16.267303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc87349-699b-4b7c-ba3f-a7eb5ebec838 service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:16.267587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc87349-699b-4b7c-ba3f-a7eb5ebec838 service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:16.267863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc87349-699b-4b7c-ba3f-a7eb5ebec838 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:16.276455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc87349-699b-4b7c-ba3f-a7eb5ebec838 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:16.278046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc87349-699b-4b7c-ba3f-a7eb5ebec838 service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:16.286464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1162/2324] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:38:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:38:16.899322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eda611-cd1f-4fcc-a435-c895ac814c80 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:16.900920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eda611-cd1f-4fcc-a435-c895ac814c80 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:16.900920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8eda611-cd1f-4fcc-a435-c895ac814c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:16.924016 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e8eda611-cd1f-4fcc-a435-c895ac814c80 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:16.928103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e8eda611-cd1f-4fcc-a435-c895ac814c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:16.935528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1163/2325] 10.222.0.20 () {58 vars in 1009 bytes} [Wed Oct 19 22:38:16 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:16.951228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc87349-699b-4b7c-ba3f-a7eb5ebec838 service swift] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:16.959109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc87349-699b-4b7c-ba3f-a7eb5ebec838 service swift] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:16.964634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cbfca5-cda3-4832-a15e-6a11cd595f68 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:16.965230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cbfca5-cda3-4832-a15e-6a11cd595f68 service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:16.965505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cbfca5-cda3-4832-a15e-6a11cd595f68 service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:16.965773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cbfca5-cda3-4832-a15e-6a11cd595f68 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:16.974655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cbfca5-cda3-4832-a15e-6a11cd595f68 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:16.976288 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cbfca5-cda3-4832-a15e-6a11cd595f68 service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:16.979805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:16.985991 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1163/2326] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:38:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:38:17.670120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9171e9-a08a-4ab0-aa3a-e4e268cd4ba4 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:17.670634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9171e9-a08a-4ab0-aa3a-e4e268cd4ba4 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:17.670745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9171e9-a08a-4ab0-aa3a-e4e268cd4ba4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:17.690094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ad9171e9-a08a-4ab0-aa3a-e4e268cd4ba4 None None] MFA Rules not processed for user `eba069a7652d4c26a6a35a73cb2be767`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:17.691390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ad9171e9-a08a-4ab0-aa3a-e4e268cd4ba4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:17.698486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1164/2327] 10.222.0.20 () {58 vars in 967 bytes} [Wed Oct 19 22:38:17 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:17.711042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cbfca5-cda3-4832-a15e-6a11cd595f68 service swift] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:17.721604 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cbfca5-cda3-4832-a15e-6a11cd595f68 service swift] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:17.726528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea90b21-6cd6-4bd7-bf31-998c853a01dd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'b19bbdff6ae1428d8cfc7a33d4fafb49', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:17.727140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea90b21-6cd6-4bd7-bf31-998c853a01dd service swift] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:17.727416 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea90b21-6cd6-4bd7-bf31-998c853a01dd service swift] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:17.727684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea90b21-6cd6-4bd7-bf31-998c853a01dd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:17.735390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea90b21-6cd6-4bd7-bf31-998c853a01dd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:17.736958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea90b21-6cd6-4bd7-bf31-998c853a01dd service swift] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:17.744797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1164/2328] 10.222.0.20 () {60 vars in 1384 bytes} [Wed Oct 19 22:38:17 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 19 22:38:20.414932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9171e9-a08a-4ab0-aa3a-e4e268cd4ba4 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:20.416396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ad9171e9-a08a-4ab0-aa3a-e4e268cd4ba4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:20.480857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9171e9-a08a-4ab0-aa3a-e4e268cd4ba4 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:20.485378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:20.486002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:20.486281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:20.486554 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] PATH_INFO: `/v3/users/bb5498b4fb7e4b54ae07bb7659a46ecd` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:20.487802 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bb5498b4fb7e4b54ae07bb7659a46ecd)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:20.489714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:20.665731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bb5498b4fb7e4b54ae07bb7659a46ecd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:20.666424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bb5498b4fb7e4b54ae07bb7659a46ecd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:20.677804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:DELETE=8 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:20.680135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bb5498b4fb7e4b54ae07bb7659a46ecd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:20.686941 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bb5498b4fb7e4b54ae07bb7659a46ecd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:20.687635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bb5498b4fb7e4b54ae07bb7659a46ecd'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:20.771910 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1165/2329] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:38:20 2022] DELETE /identity/v3/users/bb5498b4fb7e4b54ae07bb7659a46ecd => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:20.776352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0769c2a-010d-4456-815e-64119035a822 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:20.776857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0769c2a-010d-4456-815e-64119035a822 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:20.776949 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0769c2a-010d-4456-815e-64119035a822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:20.797096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a0769c2a-010d-4456-815e-64119035a822 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:20.798352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a0769c2a-010d-4456-815e-64119035a822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:20.856410 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1165/2330] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:20.865827 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:20.867317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:20.924368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4768cd8b-5e36-4e1b-ae21-b1411ba48d53 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:20.928018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f41c2a-9303-40b9-a7f8-b2ecc82549ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:20.928612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f41c2a-9303-40b9-a7f8-b2ecc82549ef service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:20.928889 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f41c2a-9303-40b9-a7f8-b2ecc82549ef service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:20.929159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f41c2a-9303-40b9-a7f8-b2ecc82549ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:20.939167 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f41c2a-9303-40b9-a7f8-b2ecc82549ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:20.940598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f41c2a-9303-40b9-a7f8-b2ecc82549ef service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:20.949916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1166/2331] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:21.203226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0769c2a-010d-4456-815e-64119035a822 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:21.204656 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a0769c2a-010d-4456-815e-64119035a822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:21.217213 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0769c2a-010d-4456-815e-64119035a822 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:21.221922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:21.222634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.222998 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.223340 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] PATH_INFO: `/v3/projects/6e4ff5b1b2104c2b8432c3bd7e749e5a` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:21.224605 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e4ff5b1b2104c2b8432c3bd7e749e5a)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:21.226862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:21.357297 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e4ff5b1b2104c2b8432c3bd7e749e5a', 'request_id': 'req-9ac25d8f-e966-4759-89cc-8fec522915c0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:21.442284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e4ff5b1b2104c2b8432c3bd7e749e5a', 'request_id': 'req-9ac25d8f-e966-4759-89cc-8fec522915c0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:21.448339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e4ff5b1b2104c2b8432c3bd7e749e5a', 'request_id': 'req-9ac25d8f-e966-4759-89cc-8fec522915c0'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:21.460217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] The token cache is being invalidate because project 6e4ff5b1b2104c2b8432c3bd7e749e5a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:38:21.468892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:21.468892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9ac25d8f-e966-4759-89cc-8fec522915c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:21.468892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1166/2332] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:38:21 2022] DELETE /identity/v3/projects/6e4ff5b1b2104c2b8432c3bd7e749e5a => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:21.469203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832a4ddc-77fe-4229-94cd-efe745139a51 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.469203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832a4ddc-77fe-4229-94cd-efe745139a51 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.469203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-832a4ddc-77fe-4229-94cd-efe745139a51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:21.487468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-832a4ddc-77fe-4229-94cd-efe745139a51 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:21.488655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-832a4ddc-77fe-4229-94cd-efe745139a51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:21.547493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1167/2333] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:21.555778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c98c9b-58c5-4b43-9b1f-7354305896a5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.556007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c98c9b-58c5-4b43-9b1f-7354305896a5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.556287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c98c9b-58c5-4b43-9b1f-7354305896a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:21.576236 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-97c98c9b-58c5-4b43-9b1f-7354305896a5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:21.581546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-97c98c9b-58c5-4b43-9b1f-7354305896a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:21.585208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1167/2334] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:21.589460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832a4ddc-77fe-4229-94cd-efe745139a51 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:21.599470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-832a4ddc-77fe-4229-94cd-efe745139a51 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:21.604151 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:21.606445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.606445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.606445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:21.606445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:38:21.606445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:21.608302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:21.619726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1168/2335] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:38:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:38:21.624526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c98c9b-58c5-4b43-9b1f-7354305896a5 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:21.633613 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c98c9b-58c5-4b43-9b1f-7354305896a5 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:21.638269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18de906-31be-462e-a112-6b401394a43a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:21.638889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18de906-31be-462e-a112-6b401394a43a admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.639195 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18de906-31be-462e-a112-6b401394a43a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.639463 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18de906-31be-462e-a112-6b401394a43a admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:21.640086 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18de906-31be-462e-a112-6b401394a43a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:21.642007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a18de906-31be-462e-a112-6b401394a43a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:21.751095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1168/2336] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:38:21 2022] POST /identity/v3/projects => generated 391 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:21.755413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:21.765368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9196bed9-97f4-49fc-9757-eb345fbb5033 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:21.821457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e6f21-e40f-4d6c-9511-edd0b84b3f5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:21.822001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e6f21-e40f-4d6c-9511-edd0b84b3f5e admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.822282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e6f21-e40f-4d6c-9511-edd0b84b3f5e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.822558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e6f21-e40f-4d6c-9511-edd0b84b3f5e admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:21.823211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5e6f21-e40f-4d6c-9511-edd0b84b3f5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:21.825135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5e6f21-e40f-4d6c-9511-edd0b84b3f5e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:21.888698 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=2,keystone:SELECT=94,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:21.894405 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1169/2337] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:38:21 2022] POST /identity/v3/users => generated 296 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:38:21.898418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18de906-31be-462e-a112-6b401394a43a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:21.909346 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18de906-31be-462e-a112-6b401394a43a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:21.914080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202c2c7f-fe28-42cb-bc49-fddb874a2956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:21.914669 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202c2c7f-fe28-42cb-bc49-fddb874a2956 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.914944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202c2c7f-fe28-42cb-bc49-fddb874a2956 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.915238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202c2c7f-fe28-42cb-bc49-fddb874a2956 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:21.915881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202c2c7f-fe28-42cb-bc49-fddb874a2956 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:21.917624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-202c2c7f-fe28-42cb-bc49-fddb874a2956 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:21.928008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1169/2338] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:21.935865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e6f21-e40f-4d6c-9511-edd0b84b3f5e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:21.945725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e6f21-e40f-4d6c-9511-edd0b84b3f5e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:21.950544 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0178991b-5561-46c2-99db-38a08c5eb523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:21.951186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178991b-5561-46c2-99db-38a08c5eb523 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.951776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178991b-5561-46c2-99db-38a08c5eb523 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.952451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0178991b-5561-46c2-99db-38a08c5eb523 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:21.953164 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0178991b-5561-46c2-99db-38a08c5eb523 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:21.954833 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0178991b-5561-46c2-99db-38a08c5eb523 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:21.965796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1170/2339] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:21.970465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202c2c7f-fe28-42cb-bc49-fddb874a2956 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:21.980285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202c2c7f-fe28-42cb-bc49-fddb874a2956 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:21.985560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:21.986191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:21.987357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:21.987357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] PATH_INFO: `/v3/projects/5ad74b51f888420b9585f8d45dc41919/users/7904662bb7be4df0a5f7bc4aba5a4492/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:22.000797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ad74b51f888420b9585f8d45dc41919, user_id=7904662bb7be4df0a5f7bc4aba5a4492, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:22.003448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:22.074292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1170/2340] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:38:21 2022] PUT /identity/v3/projects/5ad74b51f888420b9585f8d45dc41919/users/7904662bb7be4df0a5f7bc4aba5a4492/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:22.084291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe292cc-2d7a-487c-941e-1c602f5771d7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:22.084576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe292cc-2d7a-487c-941e-1c602f5771d7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:22.084852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe292cc-2d7a-487c-941e-1c602f5771d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:22.113665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3fe292cc-2d7a-487c-941e-1c602f5771d7 None None] MFA Rules not processed for user `7904662bb7be4df0a5f7bc4aba5a4492`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:22.114880 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3fe292cc-2d7a-487c-941e-1c602f5771d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:22.177607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1171/2341] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:22 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:38:22.188343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:22.190021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:22.196429 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:22.245875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64e4ce1-74a8-414d-8d01-5ca0c9ba17c7 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:22.249544 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb6864-379c-412a-9438-ba91d2c7f2f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:22.250134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb6864-379c-412a-9438-ba91d2c7f2f6 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:22.250412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb6864-379c-412a-9438-ba91d2c7f2f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:22.250708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb6864-379c-412a-9438-ba91d2c7f2f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:22.259336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fb6864-379c-412a-9438-ba91d2c7f2f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:22.260760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fb6864-379c-412a-9438-ba91d2c7f2f6 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:22.270007 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1171/2342] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:22 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:32.174736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=38 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:32.286343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=24 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:42.353491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe292cc-2d7a-487c-941e-1c602f5771d7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:42.355002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3fe292cc-2d7a-487c-941e-1c602f5771d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:42.431200 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe292cc-2d7a-487c-941e-1c602f5771d7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:42.434753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef497526-f001-4c89-90bb-e253a72de713 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:42.435696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef497526-f001-4c89-90bb-e253a72de713 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:42.435972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef497526-f001-4c89-90bb-e253a72de713 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:42.436737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef497526-f001-4c89-90bb-e253a72de713 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:42.438467 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ef497526-f001-4c89-90bb-e253a72de713 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:42.519617 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef497526-f001-4c89-90bb-e253a72de713 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:42.521087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef497526-f001-4c89-90bb-e253a72de713 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:42.533785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1172/2343] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:43.083197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb6864-379c-412a-9438-ba91d2c7f2f6 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:43.114810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb6864-379c-412a-9438-ba91d2c7f2f6 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:43.120005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:43.121312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:43.121660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:43.122140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] PATH_INFO: `/v3/users/7904662bb7be4df0a5f7bc4aba5a4492` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:43.123407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7904662bb7be4df0a5f7bc4aba5a4492)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:43.126043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:43.344379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7904662bb7be4df0a5f7bc4aba5a4492'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:43.425084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7904662bb7be4df0a5f7bc4aba5a4492'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:43.440462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7904662bb7be4df0a5f7bc4aba5a4492'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:43.449066 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7904662bb7be4df0a5f7bc4aba5a4492'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:43.449829 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7904662bb7be4df0a5f7bc4aba5a4492'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:43.457310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1172/2344] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:38:43 2022] DELETE /identity/v3/users/7904662bb7be4df0a5f7bc4aba5a4492 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:43.461816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa334e6a-0eff-4076-a0a5-ae2c3c8bcfa7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:43.462374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa334e6a-0eff-4076-a0a5-ae2c3c8bcfa7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:43.462472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa334e6a-0eff-4076-a0a5-ae2c3c8bcfa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:43.488160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-aa334e6a-0eff-4076-a0a5-ae2c3c8bcfa7 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:43.489423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-aa334e6a-0eff-4076-a0a5-ae2c3c8bcfa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:43.569683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1173/2345] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:43.594599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:43.596178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:43.684240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7598a9b2-8bcc-4855-9c09-bcb0f659cd77 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:43.690209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd886a4f-77aa-40f6-8dd5-3f52b841ad3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:43.690963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd886a4f-77aa-40f6-8dd5-3f52b841ad3b service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:43.691397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd886a4f-77aa-40f6-8dd5-3f52b841ad3b service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:43.691766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd886a4f-77aa-40f6-8dd5-3f52b841ad3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:43.706118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd886a4f-77aa-40f6-8dd5-3f52b841ad3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:43.707715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd886a4f-77aa-40f6-8dd5-3f52b841ad3b service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:43.720108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1173/2346] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:44.706421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa334e6a-0eff-4076-a0a5-ae2c3c8bcfa7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:44.708267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-aa334e6a-0eff-4076-a0a5-ae2c3c8bcfa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:44.730045 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa334e6a-0eff-4076-a0a5-ae2c3c8bcfa7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:44.740801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:44.740801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:44.741284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:44.741371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] PATH_INFO: `/v3/projects/5ad74b51f888420b9585f8d45dc41919` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:44.744743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ad74b51f888420b9585f8d45dc41919)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:44.747041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:44.758983 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=100 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:44.909187 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ad74b51f888420b9585f8d45dc41919', 'request_id': 'req-373d9f55-582e-401a-995b-bd4923ece2b0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:44.917513 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ad74b51f888420b9585f8d45dc41919', 'request_id': 'req-373d9f55-582e-401a-995b-bd4923ece2b0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:44.929084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ad74b51f888420b9585f8d45dc41919', 'request_id': 'req-373d9f55-582e-401a-995b-bd4923ece2b0'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:45.022604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] The token cache is being invalidate because project 5ad74b51f888420b9585f8d45dc41919 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:38:45.029160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:45.029160 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-373d9f55-582e-401a-995b-bd4923ece2b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:45.029674 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1174/2347] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:38:44 2022] DELETE /identity/v3/projects/5ad74b51f888420b9585f8d45dc41919 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:45.030754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af78035-1b63-42f6-a976-ef97f5a1d45a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.031442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af78035-1b63-42f6-a976-ef97f5a1d45a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.031744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af78035-1b63-42f6-a976-ef97f5a1d45a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.055293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0af78035-1b63-42f6-a976-ef97f5a1d45a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:45.064381 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0af78035-1b63-42f6-a976-ef97f5a1d45a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:45.126750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:45.148063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1174/2348] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:45.156161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d52e47-46a2-44da-b6fb-eb5c88747c1f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.156440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d52e47-46a2-44da-b6fb-eb5c88747c1f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.156708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d52e47-46a2-44da-b6fb-eb5c88747c1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.177805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b0d52e47-46a2-44da-b6fb-eb5c88747c1f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:45.179191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b0d52e47-46a2-44da-b6fb-eb5c88747c1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:45.187079 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1175/2349] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:45.191092 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af78035-1b63-42f6-a976-ef97f5a1d45a None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:45.200944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af78035-1b63-42f6-a976-ef97f5a1d45a None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:45.205617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:45.206369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.206643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.206912 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.207590 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:38:45.209395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:45.209649 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:45.222206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1175/2350] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:38:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:38:45.226676 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d52e47-46a2-44da-b6fb-eb5c88747c1f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:45.239201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d52e47-46a2-44da-b6fb-eb5c88747c1f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:45.243912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b2cbaf-9545-4754-a6fe-44fba872c5fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:45.244512 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2cbaf-9545-4754-a6fe-44fba872c5fb admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.244840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2cbaf-9545-4754-a6fe-44fba872c5fb admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.245055 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b2cbaf-9545-4754-a6fe-44fba872c5fb admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.245678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b2cbaf-9545-4754-a6fe-44fba872c5fb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:45.247691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b2cbaf-9545-4754-a6fe-44fba872c5fb admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:45.351030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1176/2351] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:38:45 2022] POST /identity/v3/projects => generated 399 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:45.355346 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:45.366753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506dfb1c-ca99-492f-b919-f2c7d8987d36 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:45.439434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca7366-2766-4c0f-8e8f-ebc512699e56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:45.440041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca7366-2766-4c0f-8e8f-ebc512699e56 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.440313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca7366-2766-4c0f-8e8f-ebc512699e56 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.440580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca7366-2766-4c0f-8e8f-ebc512699e56 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.441220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bca7366-2766-4c0f-8e8f-ebc512699e56 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:45.443119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bca7366-2766-4c0f-8e8f-ebc512699e56 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:45.540758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1176/2352] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:38:45 2022] POST /identity/v3/users => generated 300 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:45.544826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b2cbaf-9545-4754-a6fe-44fba872c5fb admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:45.557309 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b2cbaf-9545-4754-a6fe-44fba872c5fb admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:45.562638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84e306-feeb-40f3-8f09-9d275eb4ee45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:45.566534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84e306-feeb-40f3-8f09-9d275eb4ee45 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.566534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84e306-feeb-40f3-8f09-9d275eb4ee45 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.566534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84e306-feeb-40f3-8f09-9d275eb4ee45 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.566534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e84e306-feeb-40f3-8f09-9d275eb4ee45 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:45.566534 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e84e306-feeb-40f3-8f09-9d275eb4ee45 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:45.582173 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1177/2353] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:45 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:38:45.583627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca7366-2766-4c0f-8e8f-ebc512699e56 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:45.595611 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca7366-2766-4c0f-8e8f-ebc512699e56 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:45.600820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8778fab-02f6-4e16-b61d-0c9551e7f594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:45.601618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8778fab-02f6-4e16-b61d-0c9551e7f594 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.601907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8778fab-02f6-4e16-b61d-0c9551e7f594 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.602179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8778fab-02f6-4e16-b61d-0c9551e7f594 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.602831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8778fab-02f6-4e16-b61d-0c9551e7f594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:45.604509 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8778fab-02f6-4e16-b61d-0c9551e7f594 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:45.616307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1177/2354] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:45.620420 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84e306-feeb-40f3-8f09-9d275eb4ee45 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:45.630746 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84e306-feeb-40f3-8f09-9d275eb4ee45 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:45.635465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:45.636118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.636399 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.636675 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] PATH_INFO: `/v3/projects/310e09746abb4dee9d08bd319243fc71/users/8ab78faa01994abf87ed1f6456f5a91c/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.652976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=310e09746abb4dee9d08bd319243fc71, user_id=8ab78faa01994abf87ed1f6456f5a91c, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:45.656308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:45.803050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1178/2355] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:38:45 2022] PUT /identity/v3/projects/310e09746abb4dee9d08bd319243fc71/users/8ab78faa01994abf87ed1f6456f5a91c/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:38:45.814582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03c4a9e-b522-4e2e-8cd1-fbcc69e6318d None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:45.814894 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03c4a9e-b522-4e2e-8cd1-fbcc69e6318d None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:45.815199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03c4a9e-b522-4e2e-8cd1-fbcc69e6318d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:45.844834 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a03c4a9e-b522-4e2e-8cd1-fbcc69e6318d None None] MFA Rules not processed for user `8ab78faa01994abf87ed1f6456f5a91c`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:45.846052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a03c4a9e-b522-4e2e-8cd1-fbcc69e6318d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:45.914607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1178/2356] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:45 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:45.925837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:45.927617 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:46.008930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc84f90-b68a-43eb-913b-ab14e3ea5a3c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:46.013156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:46.013957 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:46.015283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:46.018919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:46.025541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:46.027281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:46.042714 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1179/2357] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:45 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:46.482234 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03c4a9e-b522-4e2e-8cd1-fbcc69e6318d None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:46.484364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a03c4a9e-b522-4e2e-8cd1-fbcc69e6318d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:46.546073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:46.567886 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03c4a9e-b522-4e2e-8cd1-fbcc69e6318d None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:46.578283 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:46.579008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:46.583547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:46.583547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:46.583547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:46.600415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:46.602025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:46.613450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1179/2358] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:46.668043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:46.669660 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:46.704202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:46.739462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ba4dcf-2663-46b2-9f94-009464e8f056 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:46.744615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:46.745252 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:46.745523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:46.745804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] PATH_INFO: `/v3/users/f9b4be7642f3441ab83cf81e1029e017` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:46.761260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f9b4be7642f3441ab83cf81e1029e017)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:46.763275 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:47.060506 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f9b4be7642f3441ab83cf81e1029e017'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.061573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f9b4be7642f3441ab83cf81e1029e017'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.073742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f9b4be7642f3441ab83cf81e1029e017'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.081974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f9b4be7642f3441ab83cf81e1029e017'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.082730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-118bc4b1-1535-4de8-9880-3cc7b0ea8a32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f9b4be7642f3441ab83cf81e1029e017'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.182866 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1180/2359] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:38:46 2022] DELETE /identity/v3/users/f9b4be7642f3441ab83cf81e1029e017 => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:47.461735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:47.471840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0f976a-1e65-466b-a20e-f67a9519d71c service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:47.529421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:47.530099 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:47.530375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:47.530643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] PATH_INFO: `/v3/projects/88309b84283a4052aa1d957047390cb8` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:47.531948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=88309b84283a4052aa1d957047390cb8)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:47.533764 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:47.652393 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88309b84283a4052aa1d957047390cb8', 'request_id': 'req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.788497 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88309b84283a4052aa1d957047390cb8', 'request_id': 'req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.794892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88309b84283a4052aa1d957047390cb8', 'request_id': 'req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.807220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] The token cache is being invalidate because project 88309b84283a4052aa1d957047390cb8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:38:47.807505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.808162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-ae5a6ffb-8ba0-41a8-9a6d-25f39426dc14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:47.809640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1180/2360] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:38:47 2022] DELETE /identity/v3/projects/88309b84283a4052aa1d957047390cb8 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:47.815271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4d0a30-20f4-4024-bb4b-06ee40b0759a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:47.815560 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4d0a30-20f4-4024-bb4b-06ee40b0759a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:47.815862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4d0a30-20f4-4024-bb4b-06ee40b0759a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:47.835102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6a4d0a30-20f4-4024-bb4b-06ee40b0759a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:47.836330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6a4d0a30-20f4-4024-bb4b-06ee40b0759a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:47.896413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1181/2361] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:47.904350 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bbac8b-8e68-4140-b94c-9d6c81793ca2 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:47.904610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bbac8b-8e68-4140-b94c-9d6c81793ca2 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:47.904893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16bbac8b-8e68-4140-b94c-9d6c81793ca2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:47.924862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-16bbac8b-8e68-4140-b94c-9d6c81793ca2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:47.926101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-16bbac8b-8e68-4140-b94c-9d6c81793ca2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:47.934150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1181/2362] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:47.938856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4d0a30-20f4-4024-bb4b-06ee40b0759a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:47.948718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4d0a30-20f4-4024-bb4b-06ee40b0759a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:47.954203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:47.955052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:47.955337 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:47.955613 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:47.956276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:38:47.956581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:47.958272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:47.974120 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1182/2363] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:38:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:38:47.975210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bbac8b-8e68-4140-b94c-9d6c81793ca2 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:47.985276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16bbac8b-8e68-4140-b94c-9d6c81793ca2 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:47.990259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f5bfc-543b-4c47-9b9f-4d9b9e51eb7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:47.990856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f5bfc-543b-4c47-9b9f-4d9b9e51eb7c admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:47.991238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f5bfc-543b-4c47-9b9f-4d9b9e51eb7c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:47.991535 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694f5bfc-543b-4c47-9b9f-4d9b9e51eb7c admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:47.992335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694f5bfc-543b-4c47-9b9f-4d9b9e51eb7c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:47.994112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694f5bfc-543b-4c47-9b9f-4d9b9e51eb7c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.101271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1182/2364] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:38:47 2022] POST /identity/v3/projects => generated 373 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:38:48.105486 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.116372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbc88a4-9bb9-493e-831d-27b102777bd1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.136491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:48.173454 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a25445-aff0-42b6-af8b-4b61a3bfb0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.174017 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a25445-aff0-42b6-af8b-4b61a3bfb0cf admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.174300 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a25445-aff0-42b6-af8b-4b61a3bfb0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.174575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a25445-aff0-42b6-af8b-4b61a3bfb0cf admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.175261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a25445-aff0-42b6-af8b-4b61a3bfb0cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.177182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a25445-aff0-42b6-af8b-4b61a3bfb0cf admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.248378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1183/2365] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:38:48 2022] POST /identity/v3/users => generated 287 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:48.252303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f5bfc-543b-4c47-9b9f-4d9b9e51eb7c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.263360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694f5bfc-543b-4c47-9b9f-4d9b9e51eb7c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.268091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eee733-69af-4087-99b6-78a4aadce2c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.268698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eee733-69af-4087-99b6-78a4aadce2c4 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.269806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eee733-69af-4087-99b6-78a4aadce2c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.269806 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eee733-69af-4087-99b6-78a4aadce2c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.269985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eee733-69af-4087-99b6-78a4aadce2c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.271503 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eee733-69af-4087-99b6-78a4aadce2c4 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.285097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1183/2366] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:48.286778 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a25445-aff0-42b6-af8b-4b61a3bfb0cf admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.303707 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a25445-aff0-42b6-af8b-4b61a3bfb0cf admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.310489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.311686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.311975 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.312254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.313280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.315415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.330046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1184/2367] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:48 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:48.332405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eee733-69af-4087-99b6-78a4aadce2c4 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.337942 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.340703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:48.343694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eee733-69af-4087-99b6-78a4aadce2c4 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.348588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bd2857-b60a-4c73-9e24-399365d80805 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.349242 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bd2857-b60a-4c73-9e24-399365d80805 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.349521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bd2857-b60a-4c73-9e24-399365d80805 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.349878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bd2857-b60a-4c73-9e24-399365d80805 admin admin] PATH_INFO: `/v3/projects/b2119bb66eff44ba92de8db500f28313/users/be083c24b1394aefbab8af2340105cc2/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.365479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bd2857-b60a-4c73-9e24-399365d80805 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2119bb66eff44ba92de8db500f28313, user_id=be083c24b1394aefbab8af2340105cc2, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.368181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34bd2857-b60a-4c73-9e24-399365d80805 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.405826 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eabf299-a859-4c8f-af53-b1bd64db8f7b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.409561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a700a811-9b80-4fcb-acd1-20098fa9b66a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.410168 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a700a811-9b80-4fcb-acd1-20098fa9b66a service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.410489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a700a811-9b80-4fcb-acd1-20098fa9b66a service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.410885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a700a811-9b80-4fcb-acd1-20098fa9b66a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.412815 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a700a811-9b80-4fcb-acd1-20098fa9b66a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:48.427136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a700a811-9b80-4fcb-acd1-20098fa9b66a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.428507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a700a811-9b80-4fcb-acd1-20098fa9b66a service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.438449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1185/2368] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:38:48.446278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1184/2369] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:38:48 2022] PUT /identity/v3/projects/b2119bb66eff44ba92de8db500f28313/users/be083c24b1394aefbab8af2340105cc2/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:48.458860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f664e4-c9ba-428c-9212-11545f926128 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.459172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f664e4-c9ba-428c-9212-11545f926128 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.459449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f664e4-c9ba-428c-9212-11545f926128 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.490616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-d6f664e4-c9ba-428c-9212-11545f926128 None None] MFA Rules not processed for user `be083c24b1394aefbab8af2340105cc2`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:48.491951 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d6f664e4-c9ba-428c-9212-11545f926128 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:48.557964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1186/2370] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:48.562645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bd2857-b60a-4c73-9e24-399365d80805 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.567284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:48.625582 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bd2857-b60a-4c73-9e24-399365d80805 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.630291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3720c-87ac-4d6a-b20d-c83b7cb78b5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.630910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3720c-87ac-4d6a-b20d-c83b7cb78b5f admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.631202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3720c-87ac-4d6a-b20d-c83b7cb78b5f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.631474 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e3720c-87ac-4d6a-b20d-c83b7cb78b5f admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.632118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e3720c-87ac-4d6a-b20d-c83b7cb78b5f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.634033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e3720c-87ac-4d6a-b20d-c83b7cb78b5f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.722364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1185/2371] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:38:48 2022] POST /identity/v3/projects => generated 371 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:48.726558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f664e4-c9ba-428c-9212-11545f926128 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.737258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f664e4-c9ba-428c-9212-11545f926128 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.770277 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:48.795956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe545e-70a2-47df-9f43-0ae9835f9efb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.796566 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe545e-70a2-47df-9f43-0ae9835f9efb admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.796843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe545e-70a2-47df-9f43-0ae9835f9efb admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.797111 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe545e-70a2-47df-9f43-0ae9835f9efb admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.797766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfe545e-70a2-47df-9f43-0ae9835f9efb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.799702 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfe545e-70a2-47df-9f43-0ae9835f9efb admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.896979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1187/2372] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:38:48 2022] POST /identity/v3/users => generated 292 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:48.898385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3720c-87ac-4d6a-b20d-c83b7cb78b5f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.909736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e3720c-87ac-4d6a-b20d-c83b7cb78b5f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.914514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d7e3e2-b421-4b5f-a4bd-d93ab9066953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.915163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d7e3e2-b421-4b5f-a4bd-d93ab9066953 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.915444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d7e3e2-b421-4b5f-a4bd-d93ab9066953 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.915715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d7e3e2-b421-4b5f-a4bd-d93ab9066953 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.916366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d7e3e2-b421-4b5f-a4bd-d93ab9066953 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.917981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09d7e3e2-b421-4b5f-a4bd-d93ab9066953 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:48.928742 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1186/2373] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:48.932886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe545e-70a2-47df-9f43-0ae9835f9efb admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:48.947329 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe545e-70a2-47df-9f43-0ae9835f9efb admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:48.953700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d694bb6-36d8-4662-a5a1-d203d7339b9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:48.954197 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d694bb6-36d8-4662-a5a1-d203d7339b9f admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:48.954479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d694bb6-36d8-4662-a5a1-d203d7339b9f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:48.954787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d694bb6-36d8-4662-a5a1-d203d7339b9f admin admin] PATH_INFO: `/v3/projects/104a66c4d054477f8b3f91df7959214e/users/586d89c13d4d4f9b83a33ad41341149d/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:48.970548 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d694bb6-36d8-4662-a5a1-d203d7339b9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=104a66c4d054477f8b3f91df7959214e, user_id=586d89c13d4d4f9b83a33ad41341149d, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:48.976563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d694bb6-36d8-4662-a5a1-d203d7339b9f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:49.054024 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1188/2374] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:38:48 2022] PUT /identity/v3/projects/104a66c4d054477f8b3f91df7959214e/users/586d89c13d4d4f9b83a33ad41341149d/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:49.058198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d7e3e2-b421-4b5f-a4bd-d93ab9066953 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:49.068399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d7e3e2-b421-4b5f-a4bd-d93ab9066953 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:49.135098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f98c164-92c4-4b99-8596-4c208af65e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:49.135752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98c164-92c4-4b99-8596-4c208af65e96 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:49.136030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98c164-92c4-4b99-8596-4c208af65e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:49.136332 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f98c164-92c4-4b99-8596-4c208af65e96 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:49.136980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f98c164-92c4-4b99-8596-4c208af65e96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:49.138639 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f98c164-92c4-4b99-8596-4c208af65e96 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:49.149910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1187/2375] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:49 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:49.154123 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d694bb6-36d8-4662-a5a1-d203d7339b9f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:49.166767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d694bb6-36d8-4662-a5a1-d203d7339b9f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:49.172133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:49.172785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:49.173067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:49.173343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] PATH_INFO: `/v3/projects/104a66c4d054477f8b3f91df7959214e/users/586d89c13d4d4f9b83a33ad41341149d/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:49.175362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=104a66c4d054477f8b3f91df7959214e, user_id=586d89c13d4d4f9b83a33ad41341149d, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:49.178018 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:49.188535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1189/2376] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:38:49 2022] PUT /identity/v3/projects/104a66c4d054477f8b3f91df7959214e/users/586d89c13d4d4f9b83a33ad41341149d/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:38:49.198258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e34f6e-fb1a-4c28-8368-68ca7ac5b066 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:49.198547 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e34f6e-fb1a-4c28-8368-68ca7ac5b066 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:49.198824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e34f6e-fb1a-4c28-8368-68ca7ac5b066 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:49.228159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-50e34f6e-fb1a-4c28-8368-68ca7ac5b066 None None] MFA Rules not processed for user `586d89c13d4d4f9b83a33ad41341149d`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:49.229321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-50e34f6e-fb1a-4c28-8368-68ca7ac5b066 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:49.294461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:49.299958 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1188/2377] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:49 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:49.311649 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:49.313339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:49.370737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109548b6-4d18-4d0b-9a97-7d5fc21039b9 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:49.374379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:49.374992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:49.375271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:49.375538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:49.384491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:49.385907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:49.395999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1190/2378] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:49 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:49.609391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e34f6e-fb1a-4c28-8368-68ca7ac5b066 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:49.633191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e34f6e-fb1a-4c28-8368-68ca7ac5b066 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:49.636899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3def8a8-9e12-4f65-809d-65239b24cbf9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:49.637722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3def8a8-9e12-4f65-809d-65239b24cbf9 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:49.638158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3def8a8-9e12-4f65-809d-65239b24cbf9 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:49.638434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3def8a8-9e12-4f65-809d-65239b24cbf9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:49.705575 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3def8a8-9e12-4f65-809d-65239b24cbf9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:49.707710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3def8a8-9e12-4f65-809d-65239b24cbf9 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:49.718325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1189/2379] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:49 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:51.981655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:51.983709 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:52.053385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153617cc-d4e5-42a0-987c-ead5fc6d8b79 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:52.058451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:52.059114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:52.059395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:52.059665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] PATH_INFO: `/v3/users/57b6c2a1a4a248158c5abbabe61ad634` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:52.075716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57b6c2a1a4a248158c5abbabe61ad634)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:52.077861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:52.101479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:52.324595 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57b6c2a1a4a248158c5abbabe61ad634'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:52.325238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57b6c2a1a4a248158c5abbabe61ad634'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:52.337480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57b6c2a1a4a248158c5abbabe61ad634'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:52.345158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57b6c2a1a4a248158c5abbabe61ad634'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:52.345792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-145d1ee4-b71f-42ec-9aa6-69ecfd949026 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57b6c2a1a4a248158c5abbabe61ad634'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:52.424129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1191/2380] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:38:51 2022] DELETE /identity/v3/users/57b6c2a1a4a248158c5abbabe61ad634 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:52.666113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3def8a8-9e12-4f65-809d-65239b24cbf9 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:52.677869 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3def8a8-9e12-4f65-809d-65239b24cbf9 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:52.736710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:52.737344 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:52.737688 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:52.737931 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] PATH_INFO: `/v3/projects/0a6de28c59e9470aa38566fdc2713ba0` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:52.739170 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a6de28c59e9470aa38566fdc2713ba0)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:52.741075 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:52.894210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a6de28c59e9470aa38566fdc2713ba0', 'request_id': 'req-73167f28-7d83-4672-905c-d2561acd1803'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:53.010546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a6de28c59e9470aa38566fdc2713ba0', 'request_id': 'req-73167f28-7d83-4672-905c-d2561acd1803'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:53.017677 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a6de28c59e9470aa38566fdc2713ba0', 'request_id': 'req-73167f28-7d83-4672-905c-d2561acd1803'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:53.029173 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] The token cache is being invalidate because project 0a6de28c59e9470aa38566fdc2713ba0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:38:53.029461 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:53.030685 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-73167f28-7d83-4672-905c-d2561acd1803 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:53.032237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1190/2381] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:38:52 2022] DELETE /identity/v3/projects/0a6de28c59e9470aa38566fdc2713ba0 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:53.037128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c4fac4-9d73-44f4-a152-c3c9f97c2782 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.037618 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c4fac4-9d73-44f4-a152-c3c9f97c2782 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.037715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c4fac4-9d73-44f4-a152-c3c9f97c2782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.057229 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b9c4fac4-9d73-44f4-a152-c3c9f97c2782 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:53.058411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b9c4fac4-9d73-44f4-a152-c3c9f97c2782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:53.124041 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1192/2382] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:38:53.131915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dec26-a114-4313-b017-706a236e6112 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.132321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dec26-a114-4313-b017-706a236e6112 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.132664 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5dec26-a114-4313-b017-706a236e6112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.159141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-ab5dec26-a114-4313-b017-706a236e6112 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:53.160435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ab5dec26-a114-4313-b017-706a236e6112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:53.172311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1191/2383] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:53.173308 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c4fac4-9d73-44f4-a152-c3c9f97c2782 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:53.186728 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c4fac4-9d73-44f4-a152-c3c9f97c2782 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:53.188908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:53.188908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.188908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.188908 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.189400 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:38:53.189706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:53.191356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:53.204020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1193/2384] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:38:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:38:53.208296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5dec26-a114-4313-b017-706a236e6112 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:53.218662 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5dec26-a114-4313-b017-706a236e6112 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:53.223456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0e23b-a37a-4907-b277-ef8fff160340 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:53.224045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0e23b-a37a-4907-b277-ef8fff160340 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.224322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0e23b-a37a-4907-b277-ef8fff160340 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.224593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd0e23b-a37a-4907-b277-ef8fff160340 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.225233 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd0e23b-a37a-4907-b277-ef8fff160340 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:53.227162 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd0e23b-a37a-4907-b277-ef8fff160340 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:53.344883 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:53.352153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1192/2385] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:38:53 2022] POST /identity/v3/projects => generated 367 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:53.356413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:53.366207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5959c95-8f65-408e-af0f-91ab9fbc24f3 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:53.422518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92f0c71-1dbe-4a82-89c8-ddc6bda28765 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:53.423883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f0c71-1dbe-4a82-89c8-ddc6bda28765 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.424161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f0c71-1dbe-4a82-89c8-ddc6bda28765 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.424457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92f0c71-1dbe-4a82-89c8-ddc6bda28765 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.425139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92f0c71-1dbe-4a82-89c8-ddc6bda28765 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:53.427107 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92f0c71-1dbe-4a82-89c8-ddc6bda28765 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:53.554509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1194/2386] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:38:53 2022] POST /identity/v3/users => generated 284 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:38:53.554844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0e23b-a37a-4907-b277-ef8fff160340 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:53.565596 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd0e23b-a37a-4907-b277-ef8fff160340 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:53.570149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e649aaa-b65e-4aa6-bd8c-de56b1efba35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:53.570760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e649aaa-b65e-4aa6-bd8c-de56b1efba35 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.571071 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e649aaa-b65e-4aa6-bd8c-de56b1efba35 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.571367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e649aaa-b65e-4aa6-bd8c-de56b1efba35 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.572042 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e649aaa-b65e-4aa6-bd8c-de56b1efba35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:53.574428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e649aaa-b65e-4aa6-bd8c-de56b1efba35 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:53.585715 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1193/2387] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:53.594372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92f0c71-1dbe-4a82-89c8-ddc6bda28765 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:53.604726 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92f0c71-1dbe-4a82-89c8-ddc6bda28765 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:53.611326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558a3922-bcd4-436c-b818-e23e671983a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:53.611928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a3922-bcd4-436c-b818-e23e671983a4 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.612202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a3922-bcd4-436c-b818-e23e671983a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.616205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558a3922-bcd4-436c-b818-e23e671983a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.616205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a3922-bcd4-436c-b818-e23e671983a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:53.616205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558a3922-bcd4-436c-b818-e23e671983a4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:53.626785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1195/2388] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:53.631510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e649aaa-b65e-4aa6-bd8c-de56b1efba35 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:53.642556 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e649aaa-b65e-4aa6-bd8c-de56b1efba35 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:53.650220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe8620f-7a06-4ae9-b751-bd5cfd4efb56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:53.650888 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe8620f-7a06-4ae9-b751-bd5cfd4efb56 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.651211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe8620f-7a06-4ae9-b751-bd5cfd4efb56 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.651480 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe8620f-7a06-4ae9-b751-bd5cfd4efb56 admin admin] PATH_INFO: `/v3/projects/7f71e1a402a042e98b16ac3c8f097370/users/f4424380d2bd4fa0bd225228e181f7ec/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.667119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe8620f-7a06-4ae9-b751-bd5cfd4efb56 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f71e1a402a042e98b16ac3c8f097370, user_id=f4424380d2bd4fa0bd225228e181f7ec, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:53.670020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfe8620f-7a06-4ae9-b751-bd5cfd4efb56 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:53.757743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1194/2389] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:38:53 2022] PUT /identity/v3/projects/7f71e1a402a042e98b16ac3c8f097370/users/f4424380d2bd4fa0bd225228e181f7ec/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:53.769280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d09d7-f20c-4579-814b-53f46ec019e6 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.769280 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d09d7-f20c-4579-814b-53f46ec019e6 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.769530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0d09d7-f20c-4579-814b-53f46ec019e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.779574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:53.812251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6e0d09d7-f20c-4579-814b-53f46ec019e6 None None] MFA Rules not processed for user `f4424380d2bd4fa0bd225228e181f7ec`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:53.813711 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6e0d09d7-f20c-4579-814b-53f46ec019e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:53.887443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1196/2390] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:53 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:53.892044 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe8620f-7a06-4ae9-b751-bd5cfd4efb56 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:53.955971 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfe8620f-7a06-4ae9-b751-bd5cfd4efb56 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:53.960591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7050bc-1541-4ef0-9699-564cd53c73bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:53.961191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7050bc-1541-4ef0-9699-564cd53c73bc admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:53.961467 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7050bc-1541-4ef0-9699-564cd53c73bc admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:53.961807 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7050bc-1541-4ef0-9699-564cd53c73bc admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:53.962451 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7050bc-1541-4ef0-9699-564cd53c73bc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:53.964650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7050bc-1541-4ef0-9699-564cd53c73bc admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:54.077538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1195/2391] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:38:53 2022] POST /identity/v3/projects => generated 367 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:54.081583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0d09d7-f20c-4579-814b-53f46ec019e6 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:54.093759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0d09d7-f20c-4579-814b-53f46ec019e6 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:54.153669 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9e604a-6df0-40c5-b40c-8153116aa5d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:54.154277 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9e604a-6df0-40c5-b40c-8153116aa5d2 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:54.155094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9e604a-6df0-40c5-b40c-8153116aa5d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:54.155368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9e604a-6df0-40c5-b40c-8153116aa5d2 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:54.155998 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9e604a-6df0-40c5-b40c-8153116aa5d2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:54.157898 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9e604a-6df0-40c5-b40c-8153116aa5d2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:54.234394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1197/2392] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:38:54 2022] POST /identity/v3/users => generated 290 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:38:54.238190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7050bc-1541-4ef0-9699-564cd53c73bc admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:54.249167 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7050bc-1541-4ef0-9699-564cd53c73bc admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:54.255063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747abd86-c7e2-4610-b770-6d8e6884a7d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:54.255698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747abd86-c7e2-4610-b770-6d8e6884a7d3 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:54.255975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747abd86-c7e2-4610-b770-6d8e6884a7d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:54.256245 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747abd86-c7e2-4610-b770-6d8e6884a7d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:54.256896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747abd86-c7e2-4610-b770-6d8e6884a7d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:54.258580 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747abd86-c7e2-4610-b770-6d8e6884a7d3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:54.270184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1196/2393] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:54 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:54.275655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9e604a-6df0-40c5-b40c-8153116aa5d2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:54.287040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9e604a-6df0-40c5-b40c-8153116aa5d2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:54.291684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffb236f-9d86-4da1-b89f-a7e2081e19ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:54.292336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb236f-9d86-4da1-b89f-a7e2081e19ce admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:54.292614 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb236f-9d86-4da1-b89f-a7e2081e19ce admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:54.292887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffb236f-9d86-4da1-b89f-a7e2081e19ce admin admin] PATH_INFO: `/v3/projects/79e5d57ed8134de99c8a0754dcffad53/users/2d826d87a82b4d3c8719577c8542f271/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:54.309730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffb236f-9d86-4da1-b89f-a7e2081e19ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=79e5d57ed8134de99c8a0754dcffad53, user_id=2d826d87a82b4d3c8719577c8542f271, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:54.313080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffb236f-9d86-4da1-b89f-a7e2081e19ce admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:54.418864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1198/2394] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:38:54 2022] PUT /identity/v3/projects/79e5d57ed8134de99c8a0754dcffad53/users/2d826d87a82b4d3c8719577c8542f271/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:54.422747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747abd86-c7e2-4610-b770-6d8e6884a7d3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:54.432800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747abd86-c7e2-4610-b770-6d8e6884a7d3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:54.489372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59fb98a-8e25-45b9-aeba-ca919717c46a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:54.489972 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59fb98a-8e25-45b9-aeba-ca919717c46a admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:54.490251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59fb98a-8e25-45b9-aeba-ca919717c46a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:54.490526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59fb98a-8e25-45b9-aeba-ca919717c46a admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:54.491198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59fb98a-8e25-45b9-aeba-ca919717c46a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:54.492799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59fb98a-8e25-45b9-aeba-ca919717c46a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:54.502756 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1197/2395] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:38:54 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:38:54.506879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffb236f-9d86-4da1-b89f-a7e2081e19ce admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:54.517494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffb236f-9d86-4da1-b89f-a7e2081e19ce admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:54.521996 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:54.522629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:54.522905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:54.523255 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] PATH_INFO: `/v3/projects/79e5d57ed8134de99c8a0754dcffad53/users/2d826d87a82b4d3c8719577c8542f271/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:54.525152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] RBAC: Authorizing `identity:create_grant(project_id=79e5d57ed8134de99c8a0754dcffad53, user_id=2d826d87a82b4d3c8719577c8542f271, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:54.527810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:54.537914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1199/2396] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:38:54 2022] PUT /identity/v3/projects/79e5d57ed8134de99c8a0754dcffad53/users/2d826d87a82b4d3c8719577c8542f271/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:54.548239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3a2780-53ab-4cb1-823f-a3b9a927d15b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:54.548526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3a2780-53ab-4cb1-823f-a3b9a927d15b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:54.548801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c3a2780-53ab-4cb1-823f-a3b9a927d15b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:54.576584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-7c3a2780-53ab-4cb1-823f-a3b9a927d15b None None] MFA Rules not processed for user `2d826d87a82b4d3c8719577c8542f271`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:54.577785 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7c3a2780-53ab-4cb1-823f-a3b9a927d15b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:54.597337 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:54.645637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1198/2397] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:54 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:54.656953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:54.658664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:54.711066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:54.716517 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26303929-ad3a-43b9-a6e3-67f42fa27afa admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:54.720489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:54.721093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:54.721371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:54.721643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:54.730842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:54.732313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:54.742404 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1200/2398] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:54 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:54.914398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3a2780-53ab-4cb1-823f-a3b9a927d15b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:54.915843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7c3a2780-53ab-4cb1-823f-a3b9a927d15b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:54.928498 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c3a2780-53ab-4cb1-823f-a3b9a927d15b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:54.932095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace85188-fba0-4864-84c3-7abd5f09e733 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:54.932679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace85188-fba0-4864-84c3-7abd5f09e733 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:54.932953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace85188-fba0-4864-84c3-7abd5f09e733 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:54.933219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace85188-fba0-4864-84c3-7abd5f09e733 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:54.989532 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace85188-fba0-4864-84c3-7abd5f09e733 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:54.990885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace85188-fba0-4864-84c3-7abd5f09e733 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:54.999631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1199/2399] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:54 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:58.224599 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:58.226052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:58.290704 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010c5ea3-d627-4e2f-9abd-d71ede79a999 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:58.295281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:58.295939 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:58.296214 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:58.296481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] PATH_INFO: `/v3/users/8ab78faa01994abf87ed1f6456f5a91c` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:58.297582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8ab78faa01994abf87ed1f6456f5a91c)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:58.299521 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:58.461787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8ab78faa01994abf87ed1f6456f5a91c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:58.462608 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8ab78faa01994abf87ed1f6456f5a91c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:58.476601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8ab78faa01994abf87ed1f6456f5a91c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:58.483712 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8ab78faa01994abf87ed1f6456f5a91c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:58.484413 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8ab78faa01994abf87ed1f6456f5a91c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:58.590334 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1201/2400] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:38:58 2022] DELETE /identity/v3/users/8ab78faa01994abf87ed1f6456f5a91c => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:38:58.595574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a089d45-2b30-4709-a796-1083a0da2c21 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:58.596062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a089d45-2b30-4709-a796-1083a0da2c21 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:58.596176 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a089d45-2b30-4709-a796-1083a0da2c21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:58.616339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-8a089d45-2b30-4709-a796-1083a0da2c21 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:38:58.617637 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8a089d45-2b30-4709-a796-1083a0da2c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:58.677750 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1200/2401] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:38:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:38:58.688263 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:58.689691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:58.748010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ee3981-8246-4cc6-8d1c-b9c0a7a49be9 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:58.752315 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:58.752918 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:58.753197 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:58.753469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:58.762348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:58.764432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:58.773759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1202/2402] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:38:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:38:59.262887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a089d45-2b30-4709-a796-1083a0da2c21 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:38:59.264698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8a089d45-2b30-4709-a796-1083a0da2c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:38:59.277061 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a089d45-2b30-4709-a796-1083a0da2c21 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:38:59.287375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:38:59.288016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:38:59.288295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:38:59.288562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] PATH_INFO: `/v3/projects/310e09746abb4dee9d08bd319243fc71` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:38:59.293259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] RBAC: Authorizing `identity:delete_project(project_id=310e09746abb4dee9d08bd319243fc71)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:38:59.308501 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:38:59.435375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '310e09746abb4dee9d08bd319243fc71', 'request_id': 'req-e31735b1-87a4-4179-b969-8cde02d5906c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:59.438226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:38:59.532981 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '310e09746abb4dee9d08bd319243fc71', 'request_id': 'req-e31735b1-87a4-4179-b969-8cde02d5906c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:59.542107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '310e09746abb4dee9d08bd319243fc71', 'request_id': 'req-e31735b1-87a4-4179-b969-8cde02d5906c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:59.554179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] The token cache is being invalidate because project 310e09746abb4dee9d08bd319243fc71 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:38:59.554469 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:59.555216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e31735b1-87a4-4179-b969-8cde02d5906c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:38:59.556697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1201/2403] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:38:59 2022] DELETE /identity/v3/projects/310e09746abb4dee9d08bd319243fc71 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:00.048348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:00.049771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:00.062407 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c3a142-4a6d-48c8-b4f1-0139f6328ea8 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:00.069916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:00.118861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:00.119514 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:00.119787 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:00.120056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] PATH_INFO: `/v3/users/f4424380d2bd4fa0bd225228e181f7ec` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:00.121156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4424380d2bd4fa0bd225228e181f7ec)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:00.123634 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:00.327575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4424380d2bd4fa0bd225228e181f7ec'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:00.328195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4424380d2bd4fa0bd225228e181f7ec'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:00.340504 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4424380d2bd4fa0bd225228e181f7ec'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:00.348185 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4424380d2bd4fa0bd225228e181f7ec'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:00.349080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4424380d2bd4fa0bd225228e181f7ec'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:00.422011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1203/2404] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:00 2022] DELETE /identity/v3/users/f4424380d2bd4fa0bd225228e181f7ec => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:00.426557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05340630-801f-435a-8dae-d5dfda120dae None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:00.427089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05340630-801f-435a-8dae-d5dfda120dae None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:00.427471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05340630-801f-435a-8dae-d5dfda120dae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:00.447445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-05340630-801f-435a-8dae-d5dfda120dae None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:00.449733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-05340630-801f-435a-8dae-d5dfda120dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:00.509766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1202/2405] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:00.520446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:00.521928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:00.582359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd1ba04-18be-4e31-ad34-76c83380e511 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:00.586142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6f51c4-009e-4451-9f7e-504fa2bb0385 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:00.586738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6f51c4-009e-4451-9f7e-504fa2bb0385 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:00.587077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6f51c4-009e-4451-9f7e-504fa2bb0385 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:00.587349 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6f51c4-009e-4451-9f7e-504fa2bb0385 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:00.599213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6f51c4-009e-4451-9f7e-504fa2bb0385 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:00.599213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6f51c4-009e-4451-9f7e-504fa2bb0385 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:00.608716 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1204/2406] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:01.009016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05340630-801f-435a-8dae-d5dfda120dae None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:01.009016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-05340630-801f-435a-8dae-d5dfda120dae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:01.021567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05340630-801f-435a-8dae-d5dfda120dae None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:01.026298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:01.027053 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:01.027504 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:01.027781 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] PATH_INFO: `/v3/projects/7f71e1a402a042e98b16ac3c8f097370` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:01.029632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f71e1a402a042e98b16ac3c8f097370)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:01.037634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:01.179823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f71e1a402a042e98b16ac3c8f097370', 'request_id': 'req-1114b829-7bd8-49e6-a06e-498017c9b466'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.250276 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f71e1a402a042e98b16ac3c8f097370', 'request_id': 'req-1114b829-7bd8-49e6-a06e-498017c9b466'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.257622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f71e1a402a042e98b16ac3c8f097370', 'request_id': 'req-1114b829-7bd8-49e6-a06e-498017c9b466'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.268676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] The token cache is being invalidate because project 7f71e1a402a042e98b16ac3c8f097370 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:01.268964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.269584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.271225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1203/2407] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:01 2022] DELETE /identity/v3/projects/7f71e1a402a042e98b16ac3c8f097370 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:01.275234 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6f51c4-009e-4451-9f7e-504fa2bb0385 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:01.284977 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6f51c4-009e-4451-9f7e-504fa2bb0385 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:01.315751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:01.341384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:01.342048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:01.342326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:01.342598 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] PATH_INFO: `/v3/users/2d826d87a82b4d3c8719577c8542f271` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:01.343749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2d826d87a82b4d3c8719577c8542f271)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:01.345730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:01.503840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2d826d87a82b4d3c8719577c8542f271'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.504488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2d826d87a82b4d3c8719577c8542f271'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.516619 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2d826d87a82b4d3c8719577c8542f271'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.523804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2d826d87a82b4d3c8719577c8542f271'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.524452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4c65d9f9-a617-4d94-b9a3-1b785aaab8f6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2d826d87a82b4d3c8719577c8542f271'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:01.618616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1205/2408] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:01 2022] DELETE /identity/v3/users/2d826d87a82b4d3c8719577c8542f271 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:02.243529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:02.244948 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:02.259374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1114b829-7bd8-49e6-a06e-498017c9b466 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:02.316164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:02.316799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:02.317080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:02.317363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] PATH_INFO: `/v3/projects/79e5d57ed8134de99c8a0754dcffad53` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:02.318558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] RBAC: Authorizing `identity:delete_project(project_id=79e5d57ed8134de99c8a0754dcffad53)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:02.320599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:02.327396 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:02.437206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '79e5d57ed8134de99c8a0754dcffad53', 'request_id': 'req-e5387967-7c0c-4c45-aaba-0bbfb9338988'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:02.510362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '79e5d57ed8134de99c8a0754dcffad53', 'request_id': 'req-e5387967-7c0c-4c45-aaba-0bbfb9338988'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:02.516355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '79e5d57ed8134de99c8a0754dcffad53', 'request_id': 'req-e5387967-7c0c-4c45-aaba-0bbfb9338988'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:02.527594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] The token cache is being invalidate because project 79e5d57ed8134de99c8a0754dcffad53 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:02.527874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:02.528541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e5387967-7c0c-4c45-aaba-0bbfb9338988 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:02.530016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1204/2409] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:02 2022] DELETE /identity/v3/projects/79e5d57ed8134de99c8a0754dcffad53 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:02.535602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9858b-7cea-4bdf-b528-a4cda24af8e7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:02.536102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9858b-7cea-4bdf-b528-a4cda24af8e7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:02.536204 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb9858b-7cea-4bdf-b528-a4cda24af8e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:02.556087 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-ddb9858b-7cea-4bdf-b528-a4cda24af8e7 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:02.557376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ddb9858b-7cea-4bdf-b528-a4cda24af8e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:02.616103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1206/2410] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:02.623626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff57a8-9cb0-4f26-8490-2c212f4ffb0f None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:02.623913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff57a8-9cb0-4f26-8490-2c212f4ffb0f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:02.624191 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ff57a8-9cb0-4f26-8490-2c212f4ffb0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:02.643682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f5ff57a8-9cb0-4f26-8490-2c212f4ffb0f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:02.644875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f5ff57a8-9cb0-4f26-8490-2c212f4ffb0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:02.653207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1205/2411] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:02.657956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9858b-7cea-4bdf-b528-a4cda24af8e7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:02.667695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb9858b-7cea-4bdf-b528-a4cda24af8e7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:02.672525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:02.680836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:02.680836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:02.680836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:02.680836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:02.680836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:02.680836 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:02.688058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1207/2412] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:39:02.692293 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff57a8-9cb0-4f26-8490-2c212f4ffb0f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:02.701748 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ff57a8-9cb0-4f26-8490-2c212f4ffb0f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:02.707104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ea0b1b-a34b-405e-ad45-f3f64ca55fcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:02.709898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea0b1b-a34b-405e-ad45-f3f64ca55fcb admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:02.709898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea0b1b-a34b-405e-ad45-f3f64ca55fcb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:02.709898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ea0b1b-a34b-405e-ad45-f3f64ca55fcb admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:02.709898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ea0b1b-a34b-405e-ad45-f3f64ca55fcb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:02.710779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ea0b1b-a34b-405e-ad45-f3f64ca55fcb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:02.768528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1206/2413] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:02 2022] POST /identity/v3/projects => generated 391 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:02.772624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:02.783768 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80099a8d-fec4-4faf-9985-a249547d23fc admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:02.821936 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:02.840576 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25da65a-1c40-4285-a20a-2f9f3044b24a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:02.841180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25da65a-1c40-4285-a20a-2f9f3044b24a admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:02.841460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25da65a-1c40-4285-a20a-2f9f3044b24a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:02.841736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b25da65a-1c40-4285-a20a-2f9f3044b24a admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:02.842375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25da65a-1c40-4285-a20a-2f9f3044b24a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:02.844358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b25da65a-1c40-4285-a20a-2f9f3044b24a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:02.932882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1208/2414] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:02 2022] POST /identity/v3/users => generated 296 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:02.937084 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ea0b1b-a34b-405e-ad45-f3f64ca55fcb admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:02.947436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ea0b1b-a34b-405e-ad45-f3f64ca55fcb admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:02.952064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98b2fd-9051-4643-a09f-15462f32c36b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:02.952652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98b2fd-9051-4643-a09f-15462f32c36b admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:02.952930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98b2fd-9051-4643-a09f-15462f32c36b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:02.953199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f98b2fd-9051-4643-a09f-15462f32c36b admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:02.953848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f98b2fd-9051-4643-a09f-15462f32c36b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:02.955996 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f98b2fd-9051-4643-a09f-15462f32c36b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:02.965995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1207/2415] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:02.970432 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25da65a-1c40-4285-a20a-2f9f3044b24a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:02.980479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b25da65a-1c40-4285-a20a-2f9f3044b24a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:02.985138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcf55fa-a122-42d5-a098-b5682d46caa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:02.987760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf55fa-a122-42d5-a098-b5682d46caa2 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:02.988814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf55fa-a122-42d5-a098-b5682d46caa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:02.988814 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcf55fa-a122-42d5-a098-b5682d46caa2 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:02.989446 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcf55fa-a122-42d5-a098-b5682d46caa2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:02.992182 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dcf55fa-a122-42d5-a098-b5682d46caa2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:03.003003 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1209/2416] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:02 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:03.007359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98b2fd-9051-4643-a09f-15462f32c36b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:03.018022 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f98b2fd-9051-4643-a09f-15462f32c36b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:03.022773 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421acf0-5bbd-41df-99ea-c4b27a0362d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:03.023406 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421acf0-5bbd-41df-99ea-c4b27a0362d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.023693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421acf0-5bbd-41df-99ea-c4b27a0362d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.023975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f421acf0-5bbd-41df-99ea-c4b27a0362d7 admin admin] PATH_INFO: `/v3/projects/c6469158f64840639a1000c43604acf1/users/1f132b33f2464269bc965f5ac7717f33/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.038837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f421acf0-5bbd-41df-99ea-c4b27a0362d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6469158f64840639a1000c43604acf1, user_id=1f132b33f2464269bc965f5ac7717f33, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:03.041536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f421acf0-5bbd-41df-99ea-c4b27a0362d7 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:03.120902 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1208/2417] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:03 2022] PUT /identity/v3/projects/c6469158f64840639a1000c43604acf1/users/1f132b33f2464269bc965f5ac7717f33/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:03.132459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265fee75-6f4e-4266-93e0-2cfb4806b282 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.132741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265fee75-6f4e-4266-93e0-2cfb4806b282 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.133020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265fee75-6f4e-4266-93e0-2cfb4806b282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.161218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-265fee75-6f4e-4266-93e0-2cfb4806b282 None None] MFA Rules not processed for user `1f132b33f2464269bc965f5ac7717f33`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:03.162429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-265fee75-6f4e-4266-93e0-2cfb4806b282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:03.231734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1210/2418] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:03 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:03.236158 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421acf0-5bbd-41df-99ea-c4b27a0362d7 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:03.296340 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f421acf0-5bbd-41df-99ea-c4b27a0362d7 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:03.300970 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c31de4-02dc-41e2-86ef-2f61e21d4948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:03.301574 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c31de4-02dc-41e2-86ef-2f61e21d4948 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.301856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c31de4-02dc-41e2-86ef-2f61e21d4948 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.302132 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c31de4-02dc-41e2-86ef-2f61e21d4948 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.302792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c31de4-02dc-41e2-86ef-2f61e21d4948 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:03.304752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c31de4-02dc-41e2-86ef-2f61e21d4948 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:03.407775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1209/2419] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:03 2022] POST /identity/v3/projects => generated 391 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:03.411817 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265fee75-6f4e-4266-93e0-2cfb4806b282 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:03.423219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265fee75-6f4e-4266-93e0-2cfb4806b282 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:03.479159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa45aa1-413a-4b49-991b-376b9cb2471f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:03.479756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa45aa1-413a-4b49-991b-376b9cb2471f admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.480033 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa45aa1-413a-4b49-991b-376b9cb2471f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.480304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa45aa1-413a-4b49-991b-376b9cb2471f admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.480952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa45aa1-413a-4b49-991b-376b9cb2471f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:03.482857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa45aa1-413a-4b49-991b-376b9cb2471f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:03.590837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1211/2420] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:03 2022] POST /identity/v3/users => generated 302 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:03.595898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c31de4-02dc-41e2-86ef-2f61e21d4948 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:03.606896 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c31de4-02dc-41e2-86ef-2f61e21d4948 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:03.616499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e83a6f-0458-4dd6-9cd1-3667e75da200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:03.617247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e83a6f-0458-4dd6-9cd1-3667e75da200 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.617609 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e83a6f-0458-4dd6-9cd1-3667e75da200 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.617969 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e83a6f-0458-4dd6-9cd1-3667e75da200 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.618691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e83a6f-0458-4dd6-9cd1-3667e75da200 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:03.620453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e83a6f-0458-4dd6-9cd1-3667e75da200 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:03.632891 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1210/2421] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:03 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:03.637110 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa45aa1-413a-4b49-991b-376b9cb2471f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:03.647285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa45aa1-413a-4b49-991b-376b9cb2471f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:03.652096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6457cf-5bcc-45a0-89c4-39b4b4929f4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:03.652738 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6457cf-5bcc-45a0-89c4-39b4b4929f4c admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.653012 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6457cf-5bcc-45a0-89c4-39b4b4929f4c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.653285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6457cf-5bcc-45a0-89c4-39b4b4929f4c admin admin] PATH_INFO: `/v3/projects/4d1fe030ad054556b05643a39559c583/users/d4bccf9415d74e4885a144c223f659c1/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.668377 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6457cf-5bcc-45a0-89c4-39b4b4929f4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d1fe030ad054556b05643a39559c583, user_id=d4bccf9415d74e4885a144c223f659c1, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:03.671063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6457cf-5bcc-45a0-89c4-39b4b4929f4c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:03.781538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1212/2422] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:03 2022] PUT /identity/v3/projects/4d1fe030ad054556b05643a39559c583/users/d4bccf9415d74e4885a144c223f659c1/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:03.786116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e83a6f-0458-4dd6-9cd1-3667e75da200 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:03.792576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:03.797658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e83a6f-0458-4dd6-9cd1-3667e75da200 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:03.853118 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3986290e-9b6a-457a-a529-f112ee9c6ce8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:03.853664 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3986290e-9b6a-457a-a529-f112ee9c6ce8 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.853924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3986290e-9b6a-457a-a529-f112ee9c6ce8 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.854189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3986290e-9b6a-457a-a529-f112ee9c6ce8 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.854822 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3986290e-9b6a-457a-a529-f112ee9c6ce8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:03.856565 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3986290e-9b6a-457a-a529-f112ee9c6ce8 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:03.867456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1211/2423] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:03 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:03.871198 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6457cf-5bcc-45a0-89c4-39b4b4929f4c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:03.881990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6457cf-5bcc-45a0-89c4-39b4b4929f4c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:03.886498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:03.887172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.887453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.887722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] PATH_INFO: `/v3/projects/4d1fe030ad054556b05643a39559c583/users/d4bccf9415d74e4885a144c223f659c1/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.889808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d1fe030ad054556b05643a39559c583, user_id=d4bccf9415d74e4885a144c223f659c1, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:03.893781 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:03.901658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:03.906428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1213/2424] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:03 2022] PUT /identity/v3/projects/4d1fe030ad054556b05643a39559c583/users/d4bccf9415d74e4885a144c223f659c1/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:03.916591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6fd9ca-c127-4e40-85e2-8f57e88a9499 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:03.917026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6fd9ca-c127-4e40-85e2-8f57e88a9499 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:03.917364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6fd9ca-c127-4e40-85e2-8f57e88a9499 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:03.948657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-0f6fd9ca-c127-4e40-85e2-8f57e88a9499 None None] MFA Rules not processed for user `d4bccf9415d74e4885a144c223f659c1`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:03.949895 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0f6fd9ca-c127-4e40-85e2-8f57e88a9499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:04.019817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1212/2425] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:03 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:04.030737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:04.032466 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:04.091074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53ec605-082d-407c-be28-8156530fea4c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:04.095176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5e26b5-558b-412c-bd24-8512335adc12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:04.095847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5e26b5-558b-412c-bd24-8512335adc12 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:04.096184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5e26b5-558b-412c-bd24-8512335adc12 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:04.096458 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5e26b5-558b-412c-bd24-8512335adc12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:04.106062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5e26b5-558b-412c-bd24-8512335adc12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:04.107611 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5e26b5-558b-412c-bd24-8512335adc12 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:04.117462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1214/2426] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:04 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:04.167214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6fd9ca-c127-4e40-85e2-8f57e88a9499 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:04.170767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-0f6fd9ca-c127-4e40-85e2-8f57e88a9499 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:04.180938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6fd9ca-c127-4e40-85e2-8f57e88a9499 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:04.184510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8a1007-e22f-4883-b1d7-e681e092a1c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:04.185098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8a1007-e22f-4883-b1d7-e681e092a1c8 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:04.185411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8a1007-e22f-4883-b1d7-e681e092a1c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:04.185680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8a1007-e22f-4883-b1d7-e681e092a1c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:04.241255 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8a1007-e22f-4883-b1d7-e681e092a1c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:04.242683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8a1007-e22f-4883-b1d7-e681e092a1c8 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:04.252104 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1213/2427] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:04 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:09.063753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5e26b5-558b-412c-bd24-8512335adc12 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:09.144916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5e26b5-558b-412c-bd24-8512335adc12 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:09.149540 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:09.150216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:09.150491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:09.150760 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] PATH_INFO: `/v3/users/1f132b33f2464269bc965f5ac7717f33` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:09.151887 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f132b33f2464269bc965f5ac7717f33)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:09.153791 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:09.350375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f132b33f2464269bc965f5ac7717f33'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:09.351070 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f132b33f2464269bc965f5ac7717f33'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:09.363960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f132b33f2464269bc965f5ac7717f33'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:09.371169 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f132b33f2464269bc965f5ac7717f33'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:09.371809 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f132b33f2464269bc965f5ac7717f33'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:09.435149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1215/2428] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:09 2022] DELETE /identity/v3/users/1f132b33f2464269bc965f5ac7717f33 => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:09.439645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dbbb6-fa41-4805-a831-5daa72ab62da None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:09.440149 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dbbb6-fa41-4805-a831-5daa72ab62da None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:09.440253 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dbbb6-fa41-4805-a831-5daa72ab62da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:09.459620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e49dbbb6-fa41-4805-a831-5daa72ab62da None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:09.462406 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e49dbbb6-fa41-4805-a831-5daa72ab62da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:09.479680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:09.520119 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1214/2429] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:09.529794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:09.531321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:09.557812 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:09.587584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd26522b-b30c-43f6-a1ac-7ae91cee396c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:09.591494 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d490707e-9e76-462f-90f7-654f35a22ccc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:09.592093 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490707e-9e76-462f-90f7-654f35a22ccc service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:09.592372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490707e-9e76-462f-90f7-654f35a22ccc service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:09.592635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d490707e-9e76-462f-90f7-654f35a22ccc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:09.602643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d490707e-9e76-462f-90f7-654f35a22ccc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:09.604211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d490707e-9e76-462f-90f7-654f35a22ccc service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:09.613706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1216/2430] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:10.076844 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49dbbb6-fa41-4805-a831-5daa72ab62da None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:10.078348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e49dbbb6-fa41-4805-a831-5daa72ab62da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:10.091493 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49dbbb6-fa41-4805-a831-5daa72ab62da None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:10.096131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:10.096986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:10.097131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:10.097307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] PATH_INFO: `/v3/projects/c6469158f64840639a1000c43604acf1` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:10.098632 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6469158f64840639a1000c43604acf1)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:10.100746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:10.210676 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6469158f64840639a1000c43604acf1', 'request_id': 'req-3e61d849-41f7-43ec-bb2e-3643030665ac'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.301096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6469158f64840639a1000c43604acf1', 'request_id': 'req-3e61d849-41f7-43ec-bb2e-3643030665ac'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.307206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6469158f64840639a1000c43604acf1', 'request_id': 'req-3e61d849-41f7-43ec-bb2e-3643030665ac'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.318922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] The token cache is being invalidate because project c6469158f64840639a1000c43604acf1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:10.319246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.319942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.321492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1215/2431] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:10 2022] DELETE /identity/v3/projects/c6469158f64840639a1000c43604acf1 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:10.325436 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d490707e-9e76-462f-90f7-654f35a22ccc service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:10.335195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d490707e-9e76-462f-90f7-654f35a22ccc service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:10.398322 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:10.399013 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:10.399324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:10.399600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] PATH_INFO: `/v3/users/d4bccf9415d74e4885a144c223f659c1` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:10.400719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d4bccf9415d74e4885a144c223f659c1)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:10.402647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:10.629613 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd4bccf9415d74e4885a144c223f659c1'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.630203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd4bccf9415d74e4885a144c223f659c1'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.642279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd4bccf9415d74e4885a144c223f659c1'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.649193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd4bccf9415d74e4885a144c223f659c1'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.649823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-43e22c00-b0b8-4d94-9e24-8ed1e67e3a2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd4bccf9415d74e4885a144c223f659c1'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:10.718086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1217/2432] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:10 2022] DELETE /identity/v3/users/d4bccf9415d74e4885a144c223f659c1 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:11.340101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:11.341528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:11.355090 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e61d849-41f7-43ec-bb2e-3643030665ac admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:11.419832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:11.420541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:11.420739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:11.420995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] PATH_INFO: `/v3/projects/4d1fe030ad054556b05643a39559c583` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:11.422120 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d1fe030ad054556b05643a39559c583)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:11.424121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:11.533142 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d1fe030ad054556b05643a39559c583', 'request_id': 'req-44da933c-8b3b-4002-abd0-3fc52bc88148'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:11.534803 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:11.629784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d1fe030ad054556b05643a39559c583', 'request_id': 'req-44da933c-8b3b-4002-abd0-3fc52bc88148'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:11.636023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d1fe030ad054556b05643a39559c583', 'request_id': 'req-44da933c-8b3b-4002-abd0-3fc52bc88148'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:11.647943 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] The token cache is being invalidate because project 4d1fe030ad054556b05643a39559c583 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:11.648232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:11.648864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-44da933c-8b3b-4002-abd0-3fc52bc88148 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:11.650343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1216/2433] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:11 2022] DELETE /identity/v3/projects/4d1fe030ad054556b05643a39559c583 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:11.655318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b067f769-c1bb-4965-9a11-45909d862f67 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:11.655794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b067f769-c1bb-4965-9a11-45909d862f67 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:11.655899 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b067f769-c1bb-4965-9a11-45909d862f67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:11.675748 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b067f769-c1bb-4965-9a11-45909d862f67 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:11.676953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b067f769-c1bb-4965-9a11-45909d862f67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:11.716773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:11.735718 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1218/2434] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:11.744434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b90481-a8eb-4e3d-99f4-760d2892e5cb None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:11.744751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b90481-a8eb-4e3d-99f4-760d2892e5cb None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:11.745028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b90481-a8eb-4e3d-99f4-760d2892e5cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:11.765216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-d7b90481-a8eb-4e3d-99f4-760d2892e5cb None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:11.766449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d7b90481-a8eb-4e3d-99f4-760d2892e5cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:11.774436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1217/2435] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:11.778429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b067f769-c1bb-4965-9a11-45909d862f67 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:11.788389 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b067f769-c1bb-4965-9a11-45909d862f67 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:11.793205 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:11.793867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:11.794115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:11.794385 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:11.795066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:11.795366 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:11.796966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:11.809238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1219/2436] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:39:11.814483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b90481-a8eb-4e3d-99f4-760d2892e5cb None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:11.823772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b90481-a8eb-4e3d-99f4-760d2892e5cb None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:11.828440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92384864-a00b-4b68-b72e-9bb59ae36a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:11.829035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92384864-a00b-4b68-b72e-9bb59ae36a4d admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:11.829317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92384864-a00b-4b68-b72e-9bb59ae36a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:11.829587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92384864-a00b-4b68-b72e-9bb59ae36a4d admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:11.830230 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92384864-a00b-4b68-b72e-9bb59ae36a4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:11.832180 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92384864-a00b-4b68-b72e-9bb59ae36a4d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:11.899336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1218/2437] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:11 2022] POST /identity/v3/projects => generated 389 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:11.903683 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:11.914030 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e69e24e-f8c7-4795-b0d5-3f39bb9907f3 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:11.970475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b68082b-3aaa-4777-a0fb-a90e1b7f3672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:11.971138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b68082b-3aaa-4777-a0fb-a90e1b7f3672 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:11.971423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b68082b-3aaa-4777-a0fb-a90e1b7f3672 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:11.971699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b68082b-3aaa-4777-a0fb-a90e1b7f3672 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:11.972365 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b68082b-3aaa-4777-a0fb-a90e1b7f3672 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:11.974316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b68082b-3aaa-4777-a0fb-a90e1b7f3672 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.046510 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1220/2438] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:11 2022] POST /identity/v3/users => generated 295 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:39:12.051349 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92384864-a00b-4b68-b72e-9bb59ae36a4d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.061630 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92384864-a00b-4b68-b72e-9bb59ae36a4d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.066239 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a1a87-c00e-4418-b1ad-43506eb99d4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.066832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a1a87-c00e-4418-b1ad-43506eb99d4f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.067150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a1a87-c00e-4418-b1ad-43506eb99d4f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.067427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498a1a87-c00e-4418-b1ad-43506eb99d4f admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.068074 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a1a87-c00e-4418-b1ad-43506eb99d4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.069682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498a1a87-c00e-4418-b1ad-43506eb99d4f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.079462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1219/2439] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:12 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:12.083338 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b68082b-3aaa-4777-a0fb-a90e1b7f3672 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.093477 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b68082b-3aaa-4777-a0fb-a90e1b7f3672 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.098541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b7d053-0ace-44b3-b7b6-9883c370f18b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.099154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7d053-0ace-44b3-b7b6-9883c370f18b admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.099435 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7d053-0ace-44b3-b7b6-9883c370f18b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.099705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b7d053-0ace-44b3-b7b6-9883c370f18b admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.100339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b7d053-0ace-44b3-b7b6-9883c370f18b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.101919 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b7d053-0ace-44b3-b7b6-9883c370f18b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.112839 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1221/2440] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:12.116604 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a1a87-c00e-4418-b1ad-43506eb99d4f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.126348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498a1a87-c00e-4418-b1ad-43506eb99d4f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.130944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776f293e-cf4b-4d82-ba5f-5f266b29477a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.131610 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776f293e-cf4b-4d82-ba5f-5f266b29477a admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.131887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776f293e-cf4b-4d82-ba5f-5f266b29477a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.132156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776f293e-cf4b-4d82-ba5f-5f266b29477a admin admin] PATH_INFO: `/v3/projects/a97c550319ba4b60a88ebcd00d364d43/users/0de09627eaf6483eb2744c50d80bd914/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.147387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776f293e-cf4b-4d82-ba5f-5f266b29477a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a97c550319ba4b60a88ebcd00d364d43, user_id=0de09627eaf6483eb2744c50d80bd914, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.150023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776f293e-cf4b-4d82-ba5f-5f266b29477a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.238593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1220/2441] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:12 2022] PUT /identity/v3/projects/a97c550319ba4b60a88ebcd00d364d43/users/0de09627eaf6483eb2744c50d80bd914/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:12.248907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eab5fdc-9120-4f7c-9dac-29556d1c2f00 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.249186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eab5fdc-9120-4f7c-9dac-29556d1c2f00 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.249455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eab5fdc-9120-4f7c-9dac-29556d1c2f00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.276391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3eab5fdc-9120-4f7c-9dac-29556d1c2f00 None None] MFA Rules not processed for user `0de09627eaf6483eb2744c50d80bd914`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:12.277575 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3eab5fdc-9120-4f7c-9dac-29556d1c2f00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:12.341067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1222/2442] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:12 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:12.345885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776f293e-cf4b-4d82-ba5f-5f266b29477a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.405653 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776f293e-cf4b-4d82-ba5f-5f266b29477a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.410809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068a492-d2ac-4aea-badc-554b47969152 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.411428 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068a492-d2ac-4aea-badc-554b47969152 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.411705 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068a492-d2ac-4aea-badc-554b47969152 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.411974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a068a492-d2ac-4aea-badc-554b47969152 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.412597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068a492-d2ac-4aea-badc-554b47969152 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.414524 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a068a492-d2ac-4aea-badc-554b47969152 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.502244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1221/2443] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:12 2022] POST /identity/v3/projects => generated 389 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:12.506166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eab5fdc-9120-4f7c-9dac-29556d1c2f00 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.517282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eab5fdc-9120-4f7c-9dac-29556d1c2f00 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.554879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:12.573066 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d7e602-19bc-4ff8-bbe8-02ded9cd316e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.573697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7e602-19bc-4ff8-bbe8-02ded9cd316e admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.573976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7e602-19bc-4ff8-bbe8-02ded9cd316e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.574281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d7e602-19bc-4ff8-bbe8-02ded9cd316e admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.574907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d7e602-19bc-4ff8-bbe8-02ded9cd316e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.576842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d7e602-19bc-4ff8-bbe8-02ded9cd316e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.708103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1223/2444] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:12 2022] POST /identity/v3/users => generated 301 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:12.711893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068a492-d2ac-4aea-badc-554b47969152 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.722910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a068a492-d2ac-4aea-badc-554b47969152 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.727591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9e5fc9-f7a9-42dd-922e-a28b0d87e117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.728217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9e5fc9-f7a9-42dd-922e-a28b0d87e117 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.728492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9e5fc9-f7a9-42dd-922e-a28b0d87e117 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.728760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e9e5fc9-f7a9-42dd-922e-a28b0d87e117 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.729458 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9e5fc9-f7a9-42dd-922e-a28b0d87e117 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.731127 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e9e5fc9-f7a9-42dd-922e-a28b0d87e117 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.741600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1222/2445] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:12.746443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d7e602-19bc-4ff8-bbe8-02ded9cd316e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.756831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d7e602-19bc-4ff8-bbe8-02ded9cd316e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.761729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9dc940-9494-4998-b123-0b8dbc0f23c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.762317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9dc940-9494-4998-b123-0b8dbc0f23c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.762600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9dc940-9494-4998-b123-0b8dbc0f23c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.762875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9dc940-9494-4998-b123-0b8dbc0f23c0 admin admin] PATH_INFO: `/v3/projects/d0f3753593d14c68a7abe100e2c4dda4/users/04f65d0dc42440988f0675f0be08cfb3/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.779769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9dc940-9494-4998-b123-0b8dbc0f23c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0f3753593d14c68a7abe100e2c4dda4, user_id=04f65d0dc42440988f0675f0be08cfb3, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.783089 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9dc940-9494-4998-b123-0b8dbc0f23c0 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.863926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1224/2446] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:12 2022] PUT /identity/v3/projects/d0f3753593d14c68a7abe100e2c4dda4/users/04f65d0dc42440988f0675f0be08cfb3/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:12.867887 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9e5fc9-f7a9-42dd-922e-a28b0d87e117 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.878752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e9e5fc9-f7a9-42dd-922e-a28b0d87e117 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.922560 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:12.934143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb856ae-be7f-4664-9c66-38b446afad02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.934743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb856ae-be7f-4664-9c66-38b446afad02 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.935055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb856ae-be7f-4664-9c66-38b446afad02 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.935330 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb856ae-be7f-4664-9c66-38b446afad02 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.936001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb856ae-be7f-4664-9c66-38b446afad02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.937599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb856ae-be7f-4664-9c66-38b446afad02 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.948517 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1223/2447] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:12 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:12.952844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9dc940-9494-4998-b123-0b8dbc0f23c0 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:12.964652 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9dc940-9494-4998-b123-0b8dbc0f23c0 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:12.969344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:12.969993 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.970297 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.970570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] PATH_INFO: `/v3/projects/d0f3753593d14c68a7abe100e2c4dda4/users/04f65d0dc42440988f0675f0be08cfb3/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:12.972582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0f3753593d14c68a7abe100e2c4dda4, user_id=04f65d0dc42440988f0675f0be08cfb3, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:12.975248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:12.985411 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1225/2448] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:12 2022] PUT /identity/v3/projects/d0f3753593d14c68a7abe100e2c4dda4/users/04f65d0dc42440988f0675f0be08cfb3/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:12.995652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe54e46-be26-4bbd-b4e2-1d20064afbdd None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:12.995939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe54e46-be26-4bbd-b4e2-1d20064afbdd None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:12.996217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe54e46-be26-4bbd-b4e2-1d20064afbdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:13.026489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-cbe54e46-be26-4bbd-b4e2-1d20064afbdd None None] MFA Rules not processed for user `04f65d0dc42440988f0675f0be08cfb3`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:13.027787 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-cbe54e46-be26-4bbd-b4e2-1d20064afbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:13.095666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1224/2449] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:12 2022] POST /identity/v3/auth/tokens => generated 3333 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:13.106261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:13.107966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:13.163582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce80fe83-1c49-467d-a197-e20fbac89dd6 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:13.167277 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd421c6c-f631-4ef6-82d7-ded8167d3ccd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:13.167860 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd421c6c-f631-4ef6-82d7-ded8167d3ccd service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:13.168132 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd421c6c-f631-4ef6-82d7-ded8167d3ccd service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:13.168397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd421c6c-f631-4ef6-82d7-ded8167d3ccd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:13.177547 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd421c6c-f631-4ef6-82d7-ded8167d3ccd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:13.178966 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd421c6c-f631-4ef6-82d7-ded8167d3ccd service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:13.189040 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1226/2450] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:13 2022] GET /identity/v3/auth/tokens => generated 3333 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:13.326004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe54e46-be26-4bbd-b4e2-1d20064afbdd None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:13.387683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe54e46-be26-4bbd-b4e2-1d20064afbdd None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:13.392394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:13.393016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:13.393295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:13.393569 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] PATH_INFO: `/v3/users/0de09627eaf6483eb2744c50d80bd914` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:13.394686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=0de09627eaf6483eb2744c50d80bd914)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:13.396647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:13.551717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0de09627eaf6483eb2744c50d80bd914'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:13.617915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0de09627eaf6483eb2744c50d80bd914'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:13.630090 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0de09627eaf6483eb2744c50d80bd914'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:13.631311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:13.637506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0de09627eaf6483eb2744c50d80bd914'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:13.638138 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0de09627eaf6483eb2744c50d80bd914'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:13.644146 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1225/2451] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:13 2022] DELETE /identity/v3/users/0de09627eaf6483eb2744c50d80bd914 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:13.648542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab94807-bc68-4a6e-8648-78d7491dd4c7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:13.648835 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab94807-bc68-4a6e-8648-78d7491dd4c7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:13.649115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab94807-bc68-4a6e-8648-78d7491dd4c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:13.668353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-4ab94807-bc68-4a6e-8648-78d7491dd4c7 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:13.669562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4ab94807-bc68-4a6e-8648-78d7491dd4c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:13.727484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1227/2452] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:13.737536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:13.738915 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:13.799470 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c11d16-326e-4b8c-8f63-951789ecc7cf admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:13.812911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5673ba1-35c0-4c18-ac94-b9d4c6c4634b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:13.812911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5673ba1-35c0-4c18-ac94-b9d4c6c4634b service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:13.812911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5673ba1-35c0-4c18-ac94-b9d4c6c4634b service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:13.812911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5673ba1-35c0-4c18-ac94-b9d4c6c4634b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:13.815736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5673ba1-35c0-4c18-ac94-b9d4c6c4634b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:13.817232 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5673ba1-35c0-4c18-ac94-b9d4c6c4634b service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:13.827100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1226/2453] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:39:14.230943 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab94807-bc68-4a6e-8648-78d7491dd4c7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:14.232373 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4ab94807-bc68-4a6e-8648-78d7491dd4c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:14.245451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab94807-bc68-4a6e-8648-78d7491dd4c7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:14.250421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:14.257677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:14.257677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:14.257677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] PATH_INFO: `/v3/projects/a97c550319ba4b60a88ebcd00d364d43` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:14.257677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a97c550319ba4b60a88ebcd00d364d43)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:14.257677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:14.374755 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:14.387947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a97c550319ba4b60a88ebcd00d364d43', 'request_id': 'req-7236f7d4-b093-4e49-b90e-24b459253d2d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.394933 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a97c550319ba4b60a88ebcd00d364d43', 'request_id': 'req-7236f7d4-b093-4e49-b90e-24b459253d2d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.400511 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a97c550319ba4b60a88ebcd00d364d43', 'request_id': 'req-7236f7d4-b093-4e49-b90e-24b459253d2d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.477195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] The token cache is being invalidate because project a97c550319ba4b60a88ebcd00d364d43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:14.477483 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.478100 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.479617 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1228/2454] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:14 2022] DELETE /identity/v3/projects/a97c550319ba4b60a88ebcd00d364d43 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:14.483545 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5673ba1-35c0-4c18-ac94-b9d4c6c4634b service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:14.493247 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5673ba1-35c0-4c18-ac94-b9d4c6c4634b service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:14.548590 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:14.549210 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:14.549507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:14.549778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] PATH_INFO: `/v3/users/04f65d0dc42440988f0675f0be08cfb3` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:14.550893 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] RBAC: Authorizing `identity:delete_user(user_id=04f65d0dc42440988f0675f0be08cfb3)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:14.552841 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:14.714220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04f65d0dc42440988f0675f0be08cfb3'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.786636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04f65d0dc42440988f0675f0be08cfb3'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.800328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04f65d0dc42440988f0675f0be08cfb3'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.807125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04f65d0dc42440988f0675f0be08cfb3'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.807769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbab4a77-d703-46e2-bb9b-0aa7b1fbc51a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04f65d0dc42440988f0675f0be08cfb3'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:14.814361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1227/2455] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:14 2022] DELETE /identity/v3/users/04f65d0dc42440988f0675f0be08cfb3 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:15.265195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:15.268166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:15.281046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7236f7d4-b093-4e49-b90e-24b459253d2d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:15.339681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:15.340327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:15.340613 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:15.340973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] PATH_INFO: `/v3/projects/d0f3753593d14c68a7abe100e2c4dda4` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:15.342084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0f3753593d14c68a7abe100e2c4dda4)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:15.344571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:15.451962 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0f3753593d14c68a7abe100e2c4dda4', 'request_id': 'req-5e592a96-8759-433a-872c-004c444d94ab'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:15.459052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0f3753593d14c68a7abe100e2c4dda4', 'request_id': 'req-5e592a96-8759-433a-872c-004c444d94ab'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:15.464944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0f3753593d14c68a7abe100e2c4dda4', 'request_id': 'req-5e592a96-8759-433a-872c-004c444d94ab'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:15.552171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] The token cache is being invalidate because project d0f3753593d14c68a7abe100e2c4dda4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:15.552390 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:15.553006 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-5e592a96-8759-433a-872c-004c444d94ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:15.554482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1229/2456] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:15 2022] DELETE /identity/v3/projects/d0f3753593d14c68a7abe100e2c4dda4 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:15.559339 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a430bf7d-4c43-4313-b648-93f2c62b42e8 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:15.559823 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a430bf7d-4c43-4313-b648-93f2c62b42e8 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:15.559937 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a430bf7d-4c43-4313-b648-93f2c62b42e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:15.580405 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a430bf7d-4c43-4313-b648-93f2c62b42e8 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:15.581631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a430bf7d-4c43-4313-b648-93f2c62b42e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:15.587050 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:15.641658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1228/2457] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:15.650382 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0159a078-b295-4d9f-b1eb-cf5dbd6ff130 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:15.650667 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0159a078-b295-4d9f-b1eb-cf5dbd6ff130 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:15.651014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0159a078-b295-4d9f-b1eb-cf5dbd6ff130 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:15.671391 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-0159a078-b295-4d9f-b1eb-cf5dbd6ff130 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:15.672633 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0159a078-b295-4d9f-b1eb-cf5dbd6ff130 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:15.680626 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1230/2458] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:15.685065 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a430bf7d-4c43-4313-b648-93f2c62b42e8 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:15.694875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a430bf7d-4c43-4313-b648-93f2c62b42e8 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:15.699468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:15.700098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:15.700379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:15.700654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:15.701307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:15.701612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:15.703250 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:15.715421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1229/2459] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:39:15.720305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0159a078-b295-4d9f-b1eb-cf5dbd6ff130 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:15.729566 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0159a078-b295-4d9f-b1eb-cf5dbd6ff130 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:15.734178 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a90e07-0010-41e9-adca-11a76210ded2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:15.734790 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a90e07-0010-41e9-adca-11a76210ded2 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:15.735112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a90e07-0010-41e9-adca-11a76210ded2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:15.735386 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5a90e07-0010-41e9-adca-11a76210ded2 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:15.736025 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a90e07-0010-41e9-adca-11a76210ded2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:15.737923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5a90e07-0010-41e9-adca-11a76210ded2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:15.857001 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1231/2460] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:15 2022] POST /identity/v3/projects => generated 385 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:15.861322 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:15.871251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1710e2-7da4-4ca4-9196-8fd53e6dc856 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:15.926634 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69010e9c-ca15-46c9-9dc6-cec9bec3e198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:15.927265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69010e9c-ca15-46c9-9dc6-cec9bec3e198 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:15.927542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69010e9c-ca15-46c9-9dc6-cec9bec3e198 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:15.927815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69010e9c-ca15-46c9-9dc6-cec9bec3e198 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:15.928452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69010e9c-ca15-46c9-9dc6-cec9bec3e198 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:15.930351 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69010e9c-ca15-46c9-9dc6-cec9bec3e198 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.011925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1230/2461] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:15 2022] POST /identity/v3/users => generated 293 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:16.016163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a90e07-0010-41e9-adca-11a76210ded2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.026274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5a90e07-0010-41e9-adca-11a76210ded2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.030989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69640829-de2d-415e-8b89-94c4eedcc6a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.032882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69640829-de2d-415e-8b89-94c4eedcc6a1 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.032882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69640829-de2d-415e-8b89-94c4eedcc6a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.032882 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69640829-de2d-415e-8b89-94c4eedcc6a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.033347 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69640829-de2d-415e-8b89-94c4eedcc6a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.034974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69640829-de2d-415e-8b89-94c4eedcc6a1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.044901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1232/2462] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:16.049599 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69010e9c-ca15-46c9-9dc6-cec9bec3e198 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.059542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69010e9c-ca15-46c9-9dc6-cec9bec3e198 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.064200 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e884db7-78ec-473d-865e-d210f83f9015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.064788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e884db7-78ec-473d-865e-d210f83f9015 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.065068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e884db7-78ec-473d-865e-d210f83f9015 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.065340 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e884db7-78ec-473d-865e-d210f83f9015 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.065988 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e884db7-78ec-473d-865e-d210f83f9015 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.067620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e884db7-78ec-473d-865e-d210f83f9015 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.077975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1231/2463] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:16.081959 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69640829-de2d-415e-8b89-94c4eedcc6a1 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.091949 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69640829-de2d-415e-8b89-94c4eedcc6a1 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.097069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb44a5f-2148-4a80-bde8-e1fa041a65f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.097641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb44a5f-2148-4a80-bde8-e1fa041a65f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.097916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb44a5f-2148-4a80-bde8-e1fa041a65f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.098186 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb44a5f-2148-4a80-bde8-e1fa041a65f7 admin admin] PATH_INFO: `/v3/projects/160d05c59e0444b3a7a8525e5b21fbe1/users/1fe094f659ac42ecb6de7b03be00b14b/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.101118 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:16.113903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb44a5f-2148-4a80-bde8-e1fa041a65f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=160d05c59e0444b3a7a8525e5b21fbe1, user_id=1fe094f659ac42ecb6de7b03be00b14b, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.116570 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb44a5f-2148-4a80-bde8-e1fa041a65f7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.211810 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1233/2464] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:16 2022] PUT /identity/v3/projects/160d05c59e0444b3a7a8525e5b21fbe1/users/1fe094f659ac42ecb6de7b03be00b14b/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:39:16.222900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634268ad-01ce-4def-9fb8-74bd9cb23136 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.223217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634268ad-01ce-4def-9fb8-74bd9cb23136 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.223520 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634268ad-01ce-4def-9fb8-74bd9cb23136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.250776 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-634268ad-01ce-4def-9fb8-74bd9cb23136 None None] MFA Rules not processed for user `1fe094f659ac42ecb6de7b03be00b14b`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:16.252034 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-634268ad-01ce-4def-9fb8-74bd9cb23136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:16.317873 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1232/2465] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:16 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:16.322546 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb44a5f-2148-4a80-bde8-e1fa041a65f7 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.384203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb44a5f-2148-4a80-bde8-e1fa041a65f7 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.388782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcae8e3-53f6-4645-8900-de00c3d5c38d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.389379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcae8e3-53f6-4645-8900-de00c3d5c38d admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.389655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcae8e3-53f6-4645-8900-de00c3d5c38d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.389923 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcae8e3-53f6-4645-8900-de00c3d5c38d admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.390551 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcae8e3-53f6-4645-8900-de00c3d5c38d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.392986 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcae8e3-53f6-4645-8900-de00c3d5c38d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.490500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1234/2466] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:16 2022] POST /identity/v3/projects => generated 385 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:16.494681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634268ad-01ce-4def-9fb8-74bd9cb23136 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.505440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634268ad-01ce-4def-9fb8-74bd9cb23136 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.509006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:16.561920 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d881cda8-cb51-4fac-9a9a-d8f55a1cbb82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.562559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d881cda8-cb51-4fac-9a9a-d8f55a1cbb82 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.563359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d881cda8-cb51-4fac-9a9a-d8f55a1cbb82 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.563639 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d881cda8-cb51-4fac-9a9a-d8f55a1cbb82 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.564264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d881cda8-cb51-4fac-9a9a-d8f55a1cbb82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.566217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d881cda8-cb51-4fac-9a9a-d8f55a1cbb82 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.703427 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1233/2467] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:16 2022] POST /identity/v3/users => generated 299 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:16.707471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcae8e3-53f6-4645-8900-de00c3d5c38d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.717892 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcae8e3-53f6-4645-8900-de00c3d5c38d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.722904 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52796c23-a1dc-4f92-9795-a2026c4871ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.723518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52796c23-a1dc-4f92-9795-a2026c4871ec admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.723792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52796c23-a1dc-4f92-9795-a2026c4871ec admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.724060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52796c23-a1dc-4f92-9795-a2026c4871ec admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.724691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52796c23-a1dc-4f92-9795-a2026c4871ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.726314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52796c23-a1dc-4f92-9795-a2026c4871ec admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.737240 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1235/2468] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:16.741317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d881cda8-cb51-4fac-9a9a-d8f55a1cbb82 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.751444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d881cda8-cb51-4fac-9a9a-d8f55a1cbb82 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.756025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61affd76-cafd-4882-a309-c950250c918d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.756747 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61affd76-cafd-4882-a309-c950250c918d admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.757016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61affd76-cafd-4882-a309-c950250c918d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.766269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61affd76-cafd-4882-a309-c950250c918d admin admin] PATH_INFO: `/v3/projects/57262fc323824299b5ca9ec29244eb39/users/52b4dc7ad4184bfba816bc553ddbc158/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.771760 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61affd76-cafd-4882-a309-c950250c918d admin admin] RBAC: Authorizing `identity:create_grant(project_id=57262fc323824299b5ca9ec29244eb39, user_id=52b4dc7ad4184bfba816bc553ddbc158, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.774397 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61affd76-cafd-4882-a309-c950250c918d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.838680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1234/2469] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:16 2022] PUT /identity/v3/projects/57262fc323824299b5ca9ec29244eb39/users/52b4dc7ad4184bfba816bc553ddbc158/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:16.842819 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52796c23-a1dc-4f92-9795-a2026c4871ec admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.853244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52796c23-a1dc-4f92-9795-a2026c4871ec admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.908313 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1523b27-07c8-4af3-8e62-0f3ba0e4f4a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.908912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1523b27-07c8-4af3-8e62-0f3ba0e4f4a1 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.909189 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1523b27-07c8-4af3-8e62-0f3ba0e4f4a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.909460 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1523b27-07c8-4af3-8e62-0f3ba0e4f4a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.910088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1523b27-07c8-4af3-8e62-0f3ba0e4f4a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.911719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1523b27-07c8-4af3-8e62-0f3ba0e4f4a1 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.921419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1236/2470] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:16 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:16.926251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61affd76-cafd-4882-a309-c950250c918d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:16.936483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61affd76-cafd-4882-a309-c950250c918d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:16.941183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:16.941820 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.942100 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.942394 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] PATH_INFO: `/v3/projects/57262fc323824299b5ca9ec29244eb39/users/52b4dc7ad4184bfba816bc553ddbc158/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.944525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] RBAC: Authorizing `identity:create_grant(project_id=57262fc323824299b5ca9ec29244eb39, user_id=52b4dc7ad4184bfba816bc553ddbc158, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:16.947212 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:16.957874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1235/2471] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:16 2022] PUT /identity/v3/projects/57262fc323824299b5ca9ec29244eb39/users/52b4dc7ad4184bfba816bc553ddbc158/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:16.969832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397c62e1-69e7-430f-bd6e-7d21f9342ae0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:16.970115 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397c62e1-69e7-430f-bd6e-7d21f9342ae0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:16.970415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397c62e1-69e7-430f-bd6e-7d21f9342ae0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:16.997271 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-397c62e1-69e7-430f-bd6e-7d21f9342ae0 None None] MFA Rules not processed for user `52b4dc7ad4184bfba816bc553ddbc158`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:16.998480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-397c62e1-69e7-430f-bd6e-7d21f9342ae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:17.048659 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:17.072536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1237/2472] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:16 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:17.083791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:17.085439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:17.141531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72427374-dd8c-44d8-bd17-0ff72447d60b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:17.145112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56497e42-a09a-47d1-b3bc-eaf07e9c74f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:17.145698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56497e42-a09a-47d1-b3bc-eaf07e9c74f3 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:17.145978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56497e42-a09a-47d1-b3bc-eaf07e9c74f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:17.146243 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56497e42-a09a-47d1-b3bc-eaf07e9c74f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:17.155015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56497e42-a09a-47d1-b3bc-eaf07e9c74f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:17.156430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56497e42-a09a-47d1-b3bc-eaf07e9c74f3 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:17.166160 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1236/2473] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:17 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:18.259930 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397c62e1-69e7-430f-bd6e-7d21f9342ae0 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:18.319824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397c62e1-69e7-430f-bd6e-7d21f9342ae0 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:18.324457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:18.330327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:18.330327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:18.330327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] PATH_INFO: `/v3/users/1fe094f659ac42ecb6de7b03be00b14b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:18.330327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1fe094f659ac42ecb6de7b03be00b14b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:18.330327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:18.472067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1fe094f659ac42ecb6de7b03be00b14b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:18.472721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1fe094f659ac42ecb6de7b03be00b14b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:18.484638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1fe094f659ac42ecb6de7b03be00b14b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:18.491672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1fe094f659ac42ecb6de7b03be00b14b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:18.492330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1fe094f659ac42ecb6de7b03be00b14b'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:18.595584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1238/2474] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:18 2022] DELETE /identity/v3/users/1fe094f659ac42ecb6de7b03be00b14b => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:39:18.600150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03380ad4-6b9a-4e90-bc2d-fc32aafe29dc None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:18.600627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03380ad4-6b9a-4e90-bc2d-fc32aafe29dc None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:18.600735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03380ad4-6b9a-4e90-bc2d-fc32aafe29dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:18.619728 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-03380ad4-6b9a-4e90-bc2d-fc32aafe29dc None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:18.620925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-03380ad4-6b9a-4e90-bc2d-fc32aafe29dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:18.678848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1237/2475] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:18.688398 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:18.689824 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:18.746703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6d8a04-13b8-4b1b-8e10-cbee557fdb3a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:18.750316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58dec59-d13c-4f1f-93f5-c7170e2501aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:18.750964 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58dec59-d13c-4f1f-93f5-c7170e2501aa service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:18.751257 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58dec59-d13c-4f1f-93f5-c7170e2501aa service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:18.751528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58dec59-d13c-4f1f-93f5-c7170e2501aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:18.760452 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58dec59-d13c-4f1f-93f5-c7170e2501aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:18.761869 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58dec59-d13c-4f1f-93f5-c7170e2501aa service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:18.771840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1239/2476] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:19.184753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03380ad4-6b9a-4e90-bc2d-fc32aafe29dc None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:19.186184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-03380ad4-6b9a-4e90-bc2d-fc32aafe29dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:19.193797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:19.202308 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03380ad4-6b9a-4e90-bc2d-fc32aafe29dc None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:19.212724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:19.212724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:19.212724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:19.212724 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] PATH_INFO: `/v3/projects/160d05c59e0444b3a7a8525e5b21fbe1` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:19.214342 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] RBAC: Authorizing `identity:delete_project(project_id=160d05c59e0444b3a7a8525e5b21fbe1)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:19.216635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:19.309338 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '160d05c59e0444b3a7a8525e5b21fbe1', 'request_id': 'req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.383147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '160d05c59e0444b3a7a8525e5b21fbe1', 'request_id': 'req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.388693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '160d05c59e0444b3a7a8525e5b21fbe1', 'request_id': 'req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.401683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] The token cache is being invalidate because project 160d05c59e0444b3a7a8525e5b21fbe1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:19.401969 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.402646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.404105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1238/2477] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:19 2022] DELETE /identity/v3/projects/160d05c59e0444b3a7a8525e5b21fbe1 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:19.408156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58dec59-d13c-4f1f-93f5-c7170e2501aa service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:19.413148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:19.418415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58dec59-d13c-4f1f-93f5-c7170e2501aa service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:19.473149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:19.473780 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:19.474332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:19.474631 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] PATH_INFO: `/v3/users/52b4dc7ad4184bfba816bc553ddbc158` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:19.475947 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] RBAC: Authorizing `identity:delete_user(user_id=52b4dc7ad4184bfba816bc553ddbc158)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:19.478332 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:19.654386 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52b4dc7ad4184bfba816bc553ddbc158'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.654988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52b4dc7ad4184bfba816bc553ddbc158'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.666823 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52b4dc7ad4184bfba816bc553ddbc158'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.674491 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52b4dc7ad4184bfba816bc553ddbc158'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.675166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-941923f1-3042-45ad-811d-8191a98febff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52b4dc7ad4184bfba816bc553ddbc158'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:19.765994 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1240/2478] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:19 2022] DELETE /identity/v3/users/52b4dc7ad4184bfba816bc553ddbc158 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:20.177364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:20.178840 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:20.193577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd8ba56-da94-4cb4-b6b8-842fa39a4899 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:20.251621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:20.252252 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:20.252553 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:20.252830 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] PATH_INFO: `/v3/projects/57262fc323824299b5ca9ec29244eb39` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:20.253978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57262fc323824299b5ca9ec29244eb39)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:20.255989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:20.372152 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57262fc323824299b5ca9ec29244eb39', 'request_id': 'req-e677f592-7e04-4888-9221-de8ae60dd256'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:20.443660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57262fc323824299b5ca9ec29244eb39', 'request_id': 'req-e677f592-7e04-4888-9221-de8ae60dd256'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:20.449447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57262fc323824299b5ca9ec29244eb39', 'request_id': 'req-e677f592-7e04-4888-9221-de8ae60dd256'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:20.460557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] The token cache is being invalidate because project 57262fc323824299b5ca9ec29244eb39 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:20.460911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:20.461542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-e677f592-7e04-4888-9221-de8ae60dd256 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:20.463333 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1239/2479] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:20 2022] DELETE /identity/v3/projects/57262fc323824299b5ca9ec29244eb39 => generated 0 bytes in 286 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:20.468302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b6ec4c-2873-41fe-922b-c825833fa8ff None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:20.468773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b6ec4c-2873-41fe-922b-c825833fa8ff None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:20.468868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b6ec4c-2873-41fe-922b-c825833fa8ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:20.489317 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-22b6ec4c-2873-41fe-922b-c825833fa8ff None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:20.490489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-22b6ec4c-2873-41fe-922b-c825833fa8ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:20.550032 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1241/2480] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:20.557792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c394e2-e310-49ea-af2b-07de64206c3f None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:20.558078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c394e2-e310-49ea-af2b-07de64206c3f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:20.558357 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c394e2-e310-49ea-af2b-07de64206c3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:20.577768 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-a1c394e2-e310-49ea-af2b-07de64206c3f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:20.578977 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a1c394e2-e310-49ea-af2b-07de64206c3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:20.586797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1240/2481] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:20.591143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b6ec4c-2873-41fe-922b-c825833fa8ff None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:20.601251 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b6ec4c-2873-41fe-922b-c825833fa8ff None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:20.605806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:20.606394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:20.606668 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:20.606939 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:20.607614 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:20.607982 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:20.609557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:20.621394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1242/2482] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:39:20.625492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c394e2-e310-49ea-af2b-07de64206c3f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:20.634759 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c394e2-e310-49ea-af2b-07de64206c3f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:20.639483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1638fa1e-1a26-4f00-929d-b8cdb6968beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:20.640069 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1638fa1e-1a26-4f00-929d-b8cdb6968beb admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:20.640347 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1638fa1e-1a26-4f00-929d-b8cdb6968beb admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:20.640619 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1638fa1e-1a26-4f00-929d-b8cdb6968beb admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:20.641259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1638fa1e-1a26-4f00-929d-b8cdb6968beb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:20.643681 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1638fa1e-1a26-4f00-929d-b8cdb6968beb admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:20.648922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:20.781821 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1241/2483] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:20 2022] POST /identity/v3/projects => generated 385 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:39:20.786472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:20.796582 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec2421-6f02-41a4-9d1b-f4fcd0f87008 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:20.801285 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:20.852609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1f9071-c722-42ff-8ade-505f92e74b7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:20.853296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f9071-c722-42ff-8ade-505f92e74b7b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:20.853574 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f9071-c722-42ff-8ade-505f92e74b7b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:20.853844 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1f9071-c722-42ff-8ade-505f92e74b7b admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:20.854461 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1f9071-c722-42ff-8ade-505f92e74b7b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:20.856423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1f9071-c722-42ff-8ade-505f92e74b7b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:20.946581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1243/2484] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:20 2022] POST /identity/v3/users => generated 293 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:20.950593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1638fa1e-1a26-4f00-929d-b8cdb6968beb admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:20.961439 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1638fa1e-1a26-4f00-929d-b8cdb6968beb admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:20.966562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6caeafa-f2ec-454a-acb2-6b483428185d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:20.967198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caeafa-f2ec-454a-acb2-6b483428185d admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:20.967510 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caeafa-f2ec-454a-acb2-6b483428185d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:20.967784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caeafa-f2ec-454a-acb2-6b483428185d admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:20.968444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6caeafa-f2ec-454a-acb2-6b483428185d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:20.970674 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6caeafa-f2ec-454a-acb2-6b483428185d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:20.980745 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1242/2485] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:20.984563 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1f9071-c722-42ff-8ade-505f92e74b7b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:20.994304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1f9071-c722-42ff-8ade-505f92e74b7b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:20.998816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf7cd9c-dd27-40cc-8628-93c0c1d1e0db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:20.999429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf7cd9c-dd27-40cc-8628-93c0c1d1e0db admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:20.999704 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf7cd9c-dd27-40cc-8628-93c0c1d1e0db admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:20.999968 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf7cd9c-dd27-40cc-8628-93c0c1d1e0db admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.000596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf7cd9c-dd27-40cc-8628-93c0c1d1e0db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:21.002170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf7cd9c-dd27-40cc-8628-93c0c1d1e0db admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:21.012444 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1244/2486] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:20 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:21.016809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6caeafa-f2ec-454a-acb2-6b483428185d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:21.027238 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6caeafa-f2ec-454a-acb2-6b483428185d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:21.031914 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3b790-9252-4a73-b04e-643002aa31ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:21.033157 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3b790-9252-4a73-b04e-643002aa31ef admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.033437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3b790-9252-4a73-b04e-643002aa31ef admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.033710 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a3b790-9252-4a73-b04e-643002aa31ef admin admin] PATH_INFO: `/v3/projects/81c829fc23d041a49c0a161ff3a5a8c6/users/f0aa0d5f82504f8e99027658e682368c/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.050073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a3b790-9252-4a73-b04e-643002aa31ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=81c829fc23d041a49c0a161ff3a5a8c6, user_id=f0aa0d5f82504f8e99027658e682368c, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:21.054348 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a3b790-9252-4a73-b04e-643002aa31ef admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:21.124150 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1243/2487] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:21 2022] PUT /identity/v3/projects/81c829fc23d041a49c0a161ff3a5a8c6/users/f0aa0d5f82504f8e99027658e682368c/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:21.134903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4831e5b0-0999-49cb-9dd8-5d4e26bd2231 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.135319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4831e5b0-0999-49cb-9dd8-5d4e26bd2231 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.135593 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4831e5b0-0999-49cb-9dd8-5d4e26bd2231 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.163050 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-4831e5b0-0999-49cb-9dd8-5d4e26bd2231 None None] MFA Rules not processed for user `f0aa0d5f82504f8e99027658e682368c`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:21.164254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4831e5b0-0999-49cb-9dd8-5d4e26bd2231 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:21.235796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1245/2488] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:21 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:21.240192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3b790-9252-4a73-b04e-643002aa31ef admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:21.300182 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a3b790-9252-4a73-b04e-643002aa31ef admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:21.304769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a134f58-e77b-4bb4-8dd7-7e426d9d32b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:21.305366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a134f58-e77b-4bb4-8dd7-7e426d9d32b3 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.305641 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a134f58-e77b-4bb4-8dd7-7e426d9d32b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.305964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a134f58-e77b-4bb4-8dd7-7e426d9d32b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.306592 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a134f58-e77b-4bb4-8dd7-7e426d9d32b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:21.308526 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a134f58-e77b-4bb4-8dd7-7e426d9d32b3 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:21.396657 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1244/2489] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:21 2022] POST /identity/v3/projects => generated 385 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:21.400665 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4831e5b0-0999-49cb-9dd8-5d4e26bd2231 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:21.411171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4831e5b0-0999-49cb-9dd8-5d4e26bd2231 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:21.467378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5becdab7-bc03-4f17-a5e4-3f6ee5c942f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:21.467972 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5becdab7-bc03-4f17-a5e4-3f6ee5c942f6 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.468250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5becdab7-bc03-4f17-a5e4-3f6ee5c942f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.468523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5becdab7-bc03-4f17-a5e4-3f6ee5c942f6 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.469155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5becdab7-bc03-4f17-a5e4-3f6ee5c942f6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:21.471273 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5becdab7-bc03-4f17-a5e4-3f6ee5c942f6 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:21.549453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1246/2490] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:21 2022] POST /identity/v3/users => generated 299 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:21.557359 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a134f58-e77b-4bb4-8dd7-7e426d9d32b3 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:21.564408 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a134f58-e77b-4bb4-8dd7-7e426d9d32b3 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:21.569028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d3cb6e-d18e-4ea4-8552-e71ee088a922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:21.569616 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d3cb6e-d18e-4ea4-8552-e71ee088a922 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.569892 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d3cb6e-d18e-4ea4-8552-e71ee088a922 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.570164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d3cb6e-d18e-4ea4-8552-e71ee088a922 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.570801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d3cb6e-d18e-4ea4-8552-e71ee088a922 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:21.572489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66d3cb6e-d18e-4ea4-8552-e71ee088a922 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:21.582704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1245/2491] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:21.586609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5becdab7-bc03-4f17-a5e4-3f6ee5c942f6 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:21.596783 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5becdab7-bc03-4f17-a5e4-3f6ee5c942f6 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:21.601453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484d3096-40e7-4e93-9dbb-008d067ad054 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:21.602088 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484d3096-40e7-4e93-9dbb-008d067ad054 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.602368 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484d3096-40e7-4e93-9dbb-008d067ad054 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.602639 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484d3096-40e7-4e93-9dbb-008d067ad054 admin admin] PATH_INFO: `/v3/projects/16bfccda51a44cb492098cef17ec1170/users/6e6b19d0625e45e3befc9352a3cf95db/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.604840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:21.618176 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484d3096-40e7-4e93-9dbb-008d067ad054 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16bfccda51a44cb492098cef17ec1170, user_id=6e6b19d0625e45e3befc9352a3cf95db, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:21.620875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484d3096-40e7-4e93-9dbb-008d067ad054 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:21.700741 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1247/2492] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:21 2022] PUT /identity/v3/projects/16bfccda51a44cb492098cef17ec1170/users/6e6b19d0625e45e3befc9352a3cf95db/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:21.704817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d3cb6e-d18e-4ea4-8552-e71ee088a922 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:21.715036 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d3cb6e-d18e-4ea4-8552-e71ee088a922 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:21.769274 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d94cdf-c854-4cfa-8038-ffca9d400d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:21.769864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d94cdf-c854-4cfa-8038-ffca9d400d9e admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.770139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d94cdf-c854-4cfa-8038-ffca9d400d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.770452 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d94cdf-c854-4cfa-8038-ffca9d400d9e admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.771108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d94cdf-c854-4cfa-8038-ffca9d400d9e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:21.772701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d94cdf-c854-4cfa-8038-ffca9d400d9e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:21.782852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1246/2493] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:21 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:39:21.786753 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484d3096-40e7-4e93-9dbb-008d067ad054 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:21.797091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484d3096-40e7-4e93-9dbb-008d067ad054 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:21.802351 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:21.803014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.803291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.803561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] PATH_INFO: `/v3/projects/16bfccda51a44cb492098cef17ec1170/users/6e6b19d0625e45e3befc9352a3cf95db/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.805447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16bfccda51a44cb492098cef17ec1170, user_id=6e6b19d0625e45e3befc9352a3cf95db, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:21.808163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:21.817952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1248/2494] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:21 2022] PUT /identity/v3/projects/16bfccda51a44cb492098cef17ec1170/users/6e6b19d0625e45e3befc9352a3cf95db/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:21.829012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2d17f3-0d6f-4f09-889f-61c610ca9c33 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.829299 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2d17f3-0d6f-4f09-889f-61c610ca9c33 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:21.829575 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c2d17f3-0d6f-4f09-889f-61c610ca9c33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:21.857375 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5c2d17f3-0d6f-4f09-889f-61c610ca9c33 None None] MFA Rules not processed for user `6e6b19d0625e45e3befc9352a3cf95db`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:21.858564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5c2d17f3-0d6f-4f09-889f-61c610ca9c33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:21.868172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:21.926179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1247/2495] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:21 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:21.936449 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:21.938261 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:21.995303 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2644a1f-4201-4f5a-9675-ac36642bbe92 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:21.998905 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2854aeb5-ee1c-403c-b9ae-ed2181381f2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:21.999520 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854aeb5-ee1c-403c-b9ae-ed2181381f2f service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:21.999793 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854aeb5-ee1c-403c-b9ae-ed2181381f2f service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:22.000058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854aeb5-ee1c-403c-b9ae-ed2181381f2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:22.053851 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2854aeb5-ee1c-403c-b9ae-ed2181381f2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:22.055305 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2854aeb5-ee1c-403c-b9ae-ed2181381f2f service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:22.063995 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1249/2496] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:21 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:27.392624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2d17f3-0d6f-4f09-889f-61c610ca9c33 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:27.402577 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c2d17f3-0d6f-4f09-889f-61c610ca9c33 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:27.406156 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:27.406864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:27.407179 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:27.407495 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:27.416512 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:27.417927 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:27.427985 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1248/2497] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:27 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:30.541324 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39f0122-27f3-4764-b998-794159f887e9 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:30.541801 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39f0122-27f3-4764-b998-794159f887e9 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:30.541895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39f0122-27f3-4764-b998-794159f887e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:30.589927 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a39f0122-27f3-4764-b998-794159f887e9 None None] MFA Rules not processed for user `690632c6d7494c22997ac41257ad32ff`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:30.591155 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a39f0122-27f3-4764-b998-794159f887e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:30.645096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:30.654920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1250/2498] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:30.965944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:30.970261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:31.027291 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ed0537-ce2c-46ef-8d05-1ddda8d50211 service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:31.030944 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fc4760-fd55-469a-b15b-4306bb97fb91 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:31.031566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fc4760-fd55-469a-b15b-4306bb97fb91 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:31.031842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fc4760-fd55-469a-b15b-4306bb97fb91 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:31.032109 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fc4760-fd55-469a-b15b-4306bb97fb91 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:31.033916 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-38fc4760-fd55-469a-b15b-4306bb97fb91 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:31.043496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fc4760-fd55-469a-b15b-4306bb97fb91 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:31.044964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38fc4760-fd55-469a-b15b-4306bb97fb91 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:31.052831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1249/2499] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:30 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:35.639059 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39f0122-27f3-4764-b998-794159f887e9 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:35.723734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39f0122-27f3-4764-b998-794159f887e9 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:35.729135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:35.729448 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:35.729723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:35.729992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] PATH_INFO: `/v3/users/f0aa0d5f82504f8e99027658e682368c` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:35.731163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0aa0d5f82504f8e99027658e682368c)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:35.735604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:35.914800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0aa0d5f82504f8e99027658e682368c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:35.915502 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0aa0d5f82504f8e99027658e682368c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:35.929201 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0aa0d5f82504f8e99027658e682368c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:35.940054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0aa0d5f82504f8e99027658e682368c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:35.940762 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0aa0d5f82504f8e99027658e682368c'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:36.032395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1251/2500] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:35 2022] DELETE /identity/v3/users/f0aa0d5f82504f8e99027658e682368c => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:36.037002 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49add071-fc04-4238-84ef-3a41fee9f990 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:36.037536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49add071-fc04-4238-84ef-3a41fee9f990 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:36.037536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49add071-fc04-4238-84ef-3a41fee9f990 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:36.065925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-49add071-fc04-4238-84ef-3a41fee9f990 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:36.065925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-49add071-fc04-4238-84ef-3a41fee9f990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:36.126441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1250/2501] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:36.137101 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:36.138647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:36.198010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37f4b26-8adc-4170-b171-1ce784eb74e7 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:36.201740 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:36.202350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:36.202628 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:36.202925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:36.213135 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:36.214569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:36.224330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1252/2502] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:36.872936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49add071-fc04-4238-84ef-3a41fee9f990 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:36.874398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-49add071-fc04-4238-84ef-3a41fee9f990 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:36.886555 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49add071-fc04-4238-84ef-3a41fee9f990 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:36.890194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:36.890816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:36.891127 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:36.891399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:36.893078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:36.902426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:36.903903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:36.912295 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1251/2503] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:36.955336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:36.957194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:36.974267 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc53f563-4124-412c-a076-98a430a55971 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:36.980469 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:36.980856 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:36.981237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:36.981639 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] PATH_INFO: `/v3/projects/81c829fc23d041a49c0a161ff3a5a8c6` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:36.982883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] RBAC: Authorizing `identity:delete_project(project_id=81c829fc23d041a49c0a161ff3a5a8c6)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:36.985184 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:36.998992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:37.120245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81c829fc23d041a49c0a161ff3a5a8c6', 'request_id': 'req-a75992c0-5245-4bef-9f99-489e8ed5936e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.128804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81c829fc23d041a49c0a161ff3a5a8c6', 'request_id': 'req-a75992c0-5245-4bef-9f99-489e8ed5936e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.135730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81c829fc23d041a49c0a161ff3a5a8c6', 'request_id': 'req-a75992c0-5245-4bef-9f99-489e8ed5936e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.261330 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] The token cache is being invalidate because project 81c829fc23d041a49c0a161ff3a5a8c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:37.261745 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.262507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.264731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1253/2504] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:36 2022] DELETE /identity/v3/projects/81c829fc23d041a49c0a161ff3a5a8c6 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:37.268453 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:37.293525 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf8e380-0c7d-46e7-b1e0-05666f7d4d53 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:37.315825 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:37.356106 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:37.356732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:37.357224 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:37.357647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] PATH_INFO: `/v3/users/6e6b19d0625e45e3befc9352a3cf95db` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:37.359606 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e6b19d0625e45e3befc9352a3cf95db)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:37.361721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:37.576206 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e6b19d0625e45e3befc9352a3cf95db'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.645080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e6b19d0625e45e3befc9352a3cf95db'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.658652 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e6b19d0625e45e3befc9352a3cf95db'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.665853 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e6b19d0625e45e3befc9352a3cf95db'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.666576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-897bec22-02ba-4f47-a4f9-28980b0e81df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e6b19d0625e45e3befc9352a3cf95db'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:37.672910 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1252/2505] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:37 2022] DELETE /identity/v3/users/6e6b19d0625e45e3befc9352a3cf95db => generated 0 bytes in 405 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:38.117731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:38.119948 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:38.140586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a75992c0-5245-4bef-9f99-489e8ed5936e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:38.202171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:38.202811 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:38.203125 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:38.203451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] PATH_INFO: `/v3/projects/16bfccda51a44cb492098cef17ec1170` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:38.204581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] RBAC: Authorizing `identity:delete_project(project_id=16bfccda51a44cb492098cef17ec1170)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:38.206495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:38.325473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16bfccda51a44cb492098cef17ec1170', 'request_id': 'req-4056791d-4afe-415f-9d5c-3a7b842ba7fe'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:38.332237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16bfccda51a44cb492098cef17ec1170', 'request_id': 'req-4056791d-4afe-415f-9d5c-3a7b842ba7fe'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:38.338672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16bfccda51a44cb492098cef17ec1170', 'request_id': 'req-4056791d-4afe-415f-9d5c-3a7b842ba7fe'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:38.437361 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] The token cache is being invalidate because project 16bfccda51a44cb492098cef17ec1170 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:38.437722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:38.438473 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4056791d-4afe-415f-9d5c-3a7b842ba7fe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:38.440048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1254/2506] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:38 2022] DELETE /identity/v3/projects/16bfccda51a44cb492098cef17ec1170 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:38.446481 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223455c5-b5c9-44a1-963b-bb54176e147f None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:38.447023 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223455c5-b5c9-44a1-963b-bb54176e147f None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:38.447144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223455c5-b5c9-44a1-963b-bb54176e147f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:38.467508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-223455c5-b5c9-44a1-963b-bb54176e147f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:38.468793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-223455c5-b5c9-44a1-963b-bb54176e147f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:38.541680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1253/2507] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:38.542775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd0deb4-85b4-4284-ae2c-4622f0c979f5 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:38.543097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd0deb4-85b4-4284-ae2c-4622f0c979f5 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:38.543378 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcd0deb4-85b4-4284-ae2c-4622f0c979f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:38.563981 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-dcd0deb4-85b4-4284-ae2c-4622f0c979f5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:38.565202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-dcd0deb4-85b4-4284-ae2c-4622f0c979f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:38.573807 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1255/2508] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:38.578019 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223455c5-b5c9-44a1-963b-bb54176e147f None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:38.588438 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223455c5-b5c9-44a1-963b-bb54176e147f None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:38.593226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:38.594041 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:38.594327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:38.594603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:38.595310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:38.595617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:38.597813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:38.610226 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1254/2509] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:39:38.614747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd0deb4-85b4-4284-ae2c-4622f0c979f5 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:38.624369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcd0deb4-85b4-4284-ae2c-4622f0c979f5 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:38.629166 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793e7c91-bf78-4087-9af4-80a897f1da71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:38.629769 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793e7c91-bf78-4087-9af4-80a897f1da71 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:38.630046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793e7c91-bf78-4087-9af4-80a897f1da71 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:38.630319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793e7c91-bf78-4087-9af4-80a897f1da71 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:38.630978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793e7c91-bf78-4087-9af4-80a897f1da71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:38.632906 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793e7c91-bf78-4087-9af4-80a897f1da71 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:38.744948 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:38.754552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1256/2510] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:38 2022] POST /identity/v3/projects => generated 385 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:38.755795 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:38.765881 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39459e66-4294-432c-b851-606b6de69fc5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:38.802698 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:38.835979 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2119336b-5468-4af0-ab03-364c3e5cbe71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:38.836590 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2119336b-5468-4af0-ab03-364c3e5cbe71 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:38.836845 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2119336b-5468-4af0-ab03-364c3e5cbe71 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:38.837114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2119336b-5468-4af0-ab03-364c3e5cbe71 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:38.837744 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2119336b-5468-4af0-ab03-364c3e5cbe71 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:38.840114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2119336b-5468-4af0-ab03-364c3e5cbe71 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:38.926319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1255/2511] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:38 2022] POST /identity/v3/users => generated 293 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:38.930355 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793e7c91-bf78-4087-9af4-80a897f1da71 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:38.941410 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793e7c91-bf78-4087-9af4-80a897f1da71 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:38.946426 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cd271e-50bb-4914-9736-513b479496d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:38.947129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cd271e-50bb-4914-9736-513b479496d2 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:38.947526 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cd271e-50bb-4914-9736-513b479496d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:38.947808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cd271e-50bb-4914-9736-513b479496d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:38.948447 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cd271e-50bb-4914-9736-513b479496d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:38.950052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cd271e-50bb-4914-9736-513b479496d2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:38.960572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1257/2512] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:38.964797 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2119336b-5468-4af0-ab03-364c3e5cbe71 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:38.975603 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2119336b-5468-4af0-ab03-364c3e5cbe71 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:38.980523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aebb580-6c96-4621-b68d-8cd686c06813 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:38.981131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aebb580-6c96-4621-b68d-8cd686c06813 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:38.981668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aebb580-6c96-4621-b68d-8cd686c06813 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:38.982043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aebb580-6c96-4621-b68d-8cd686c06813 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:38.982709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aebb580-6c96-4621-b68d-8cd686c06813 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:38.984362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aebb580-6c96-4621-b68d-8cd686c06813 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:38.997626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1256/2513] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:38.998776 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cd271e-50bb-4914-9736-513b479496d2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:39.009764 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cd271e-50bb-4914-9736-513b479496d2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:39.014586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:39.015424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.015806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.016034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] PATH_INFO: `/v3/projects/e305119e1a694fdf8387316a9cea1067/users/0547b4bc7c04464698958567ead51b43/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.032049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e305119e1a694fdf8387316a9cea1067, user_id=0547b4bc7c04464698958567ead51b43, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:39.034710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:39.135076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1258/2514] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:38 2022] PUT /identity/v3/projects/e305119e1a694fdf8387316a9cea1067/users/0547b4bc7c04464698958567ead51b43/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:39.147004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8230e70-4d97-4bda-a141-87efb9335f65 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.147298 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8230e70-4d97-4bda-a141-87efb9335f65 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.147576 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8230e70-4d97-4bda-a141-87efb9335f65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.175679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-e8230e70-4d97-4bda-a141-87efb9335f65 None None] MFA Rules not processed for user `0547b4bc7c04464698958567ead51b43`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:39.176911 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e8230e70-4d97-4bda-a141-87efb9335f65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:39.244316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1257/2515] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:39 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:39.256029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:39.257752 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:39.313967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c37ba86-1e37-4c5d-a17a-f83ddd9b7e53 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:39.318625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866e1e06-b403-4d28-87cf-8c9b4c9b5834 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:39.319010 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e1e06-b403-4d28-87cf-8c9b4c9b5834 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.319290 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e1e06-b403-4d28-87cf-8c9b4c9b5834 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.319561 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866e1e06-b403-4d28-87cf-8c9b4c9b5834 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.329771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8230e70-4d97-4bda-a141-87efb9335f65 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:39.331126 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866e1e06-b403-4d28-87cf-8c9b4c9b5834 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:39.331269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-e8230e70-4d97-4bda-a141-87efb9335f65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:39.332597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-866e1e06-b403-4d28-87cf-8c9b4c9b5834 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:39.342072 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1259/2516] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:39 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:39.360784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866e1e06-b403-4d28-87cf-8c9b4c9b5834 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:39.399115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8230e70-4d97-4bda-a141-87efb9335f65 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:39.403889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:39.404597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.404956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.405299 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] PATH_INFO: `/v3/users/be083c24b1394aefbab8af2340105cc2` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.406536 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=be083c24b1394aefbab8af2340105cc2)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:39.408573 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:39.422725 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866e1e06-b403-4d28-87cf-8c9b4c9b5834 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:39.428037 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:39.428662 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.428944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.429219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] PATH_INFO: `/v3/users/0547b4bc7c04464698958567ead51b43` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.430358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0547b4bc7c04464698958567ead51b43)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:39.432312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:39.584378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'be083c24b1394aefbab8af2340105cc2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.630974 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0547b4bc7c04464698958567ead51b43'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.631650 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0547b4bc7c04464698958567ead51b43'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.643371 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0547b4bc7c04464698958567ead51b43'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.648539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'be083c24b1394aefbab8af2340105cc2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.650535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0547b4bc7c04464698958567ead51b43'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.651270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-2f409d32-77fb-4a4e-9963-3b3f741e0e0e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0547b4bc7c04464698958567ead51b43'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.661040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'be083c24b1394aefbab8af2340105cc2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.667769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'be083c24b1394aefbab8af2340105cc2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.668426 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c9fc2e5-fdda-45ba-b348-f4eb84b6dec4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'be083c24b1394aefbab8af2340105cc2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:39.674796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1258/2517] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:39 2022] DELETE /identity/v3/users/be083c24b1394aefbab8af2340105cc2 => generated 0 bytes in 346 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:39:39.679646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f682177-d7e8-4aa0-b733-2fe5017c9cb4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.679966 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f682177-d7e8-4aa0-b733-2fe5017c9cb4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.686701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f682177-d7e8-4aa0-b733-2fe5017c9cb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.700484 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-5f682177-d7e8-4aa0-b733-2fe5017c9cb4 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:39.701790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5f682177-d7e8-4aa0-b733-2fe5017c9cb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:39.737611 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1260/2518] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:39 2022] DELETE /identity/v3/users/0547b4bc7c04464698958567ead51b43 => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:39:39.742737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017b45ab-26ab-4c1a-98c6-9185c0f4094b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.743369 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017b45ab-26ab-4c1a-98c6-9185c0f4094b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.743832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017b45ab-26ab-4c1a-98c6-9185c0f4094b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.747727 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:INSERT=1,keystone:DELETE=9 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:39.760313 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1259/2519] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:39.765639 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-017b45ab-26ab-4c1a-98c6-9185c0f4094b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:39.766837 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-017b45ab-26ab-4c1a-98c6-9185c0f4094b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:39.773258 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f682177-d7e8-4aa0-b733-2fe5017c9cb4 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:39.774683 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5f682177-d7e8-4aa0-b733-2fe5017c9cb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:39.785953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:INSERT=3,keystone:DELETE=6 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:39.832149 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1261/2520] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:39.834184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f682177-d7e8-4aa0-b733-2fe5017c9cb4 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:39.843408 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017b45ab-26ab-4c1a-98c6-9185c0f4094b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:39.886521 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:39.887161 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.887435 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.887700 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.897317 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:39.898767 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:39.912340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017b45ab-26ab-4c1a-98c6-9185c0f4094b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:39.912704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1260/2521] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:39:39.912806 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:39.913295 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:39.913573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:39.913841 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:39.923211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:39.924795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:39.935509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1262/2522] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:39:40.476398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:40.476678 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:40.477906 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:40.478713 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:40.492068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f80376-1fe6-4503-9277-5fa316060c43 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:40.497612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:40.498078 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:40.498289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:40.498645 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] PATH_INFO: `/v3/projects/e305119e1a694fdf8387316a9cea1067` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:40.499973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e305119e1a694fdf8387316a9cea1067)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:40.502144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:40.513058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a842265-74e9-4504-8f3e-6d9be27b9df0 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:40.518341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:40.518465 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:40.519635 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:40.519917 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:40.520193 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] PATH_INFO: `/v3/projects/b2119bb66eff44ba92de8db500f28313` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:40.521360 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b2119bb66eff44ba92de8db500f28313)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:40.523293 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:40.631441 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e305119e1a694fdf8387316a9cea1067', 'request_id': 'req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.655081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b2119bb66eff44ba92de8db500f28313', 'request_id': 'req-4444a8e8-6ef9-47db-82b9-62575f125d5d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.661359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b2119bb66eff44ba92de8db500f28313', 'request_id': 'req-4444a8e8-6ef9-47db-82b9-62575f125d5d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.666862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b2119bb66eff44ba92de8db500f28313', 'request_id': 'req-4444a8e8-6ef9-47db-82b9-62575f125d5d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.694285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e305119e1a694fdf8387316a9cea1067', 'request_id': 'req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.700184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e305119e1a694fdf8387316a9cea1067', 'request_id': 'req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.712278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] The token cache is being invalidate because project e305119e1a694fdf8387316a9cea1067 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:40.713211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.713866 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-9b6b99cf-80a4-4028-8a84-5c8a9964d3a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.715360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1261/2523] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:40 2022] DELETE /identity/v3/projects/e305119e1a694fdf8387316a9cea1067 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:40.720956 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264d2796-7a9f-4f0f-9ac6-0d82797d000a None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:40.721184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264d2796-7a9f-4f0f-9ac6-0d82797d000a None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:40.721456 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264d2796-7a9f-4f0f-9ac6-0d82797d000a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:40.740383 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-264d2796-7a9f-4f0f-9ac6-0d82797d000a None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:40.741587 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-264d2796-7a9f-4f0f-9ac6-0d82797d000a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:40.751147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] The token cache is being invalidate because project b2119bb66eff44ba92de8db500f28313 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:40.751684 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.752057 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:40.754172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1263/2524] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:40 2022] DELETE /identity/v3/projects/b2119bb66eff44ba92de8db500f28313 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:40.758170 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:40.759949 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:40.775281 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:INSERT=2,keystone:DELETE=8 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:40.804709 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1262/2525] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:40.813205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16733c1-f950-4eb0-aafb-e60145314b43 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:40.813494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16733c1-f950-4eb0-aafb-e60145314b43 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:40.818285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16733c1-f950-4eb0-aafb-e60145314b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:40.827573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4444a8e8-6ef9-47db-82b9-62575f125d5d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:40.832248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:40.832871 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:40.833150 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:40.833423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] PATH_INFO: `/v3/users/586d89c13d4d4f9b83a33ad41341149d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:40.833549 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-d16733c1-f950-4eb0-aafb-e60145314b43 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:40.834573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=586d89c13d4d4f9b83a33ad41341149d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:40.834691 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d16733c1-f950-4eb0-aafb-e60145314b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:40.836616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:40.843064 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1263/2526] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:40.847095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16733c1-f950-4eb0-aafb-e60145314b43 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:40.856172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16733c1-f950-4eb0-aafb-e60145314b43 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:40.861133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:40.861682 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:40.861953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:40.862221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:40.863404 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:40.863719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:40.865306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:40.876694 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1264/2527] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:39:40.881554 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:40.890824 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1eaed3-da01-4d21-bd65-40bed7e44044 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:40.895319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52262b1-eb7c-42f5-915a-3f5c9b1c593d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:40.895868 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52262b1-eb7c-42f5-915a-3f5c9b1c593d admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:40.896143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52262b1-eb7c-42f5-915a-3f5c9b1c593d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:40.896412 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c52262b1-eb7c-42f5-915a-3f5c9b1c593d admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:40.897015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52262b1-eb7c-42f5-915a-3f5c9b1c593d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:40.898849 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c52262b1-eb7c-42f5-915a-3f5c9b1c593d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:40.992913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1265/2528] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:40 2022] POST /identity/v3/projects => generated 373 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:39:40.997077 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52262b1-eb7c-42f5-915a-3f5c9b1c593d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:41.057417 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c52262b1-eb7c-42f5-915a-3f5c9b1c593d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:41.062208 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:41.062798 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:41.063108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:41.063379 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:41.064014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:41.065145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '586d89c13d4d4f9b83a33ad41341149d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.065893 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '586d89c13d4d4f9b83a33ad41341149d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.066010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:41.074476 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:41.078805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '586d89c13d4d4f9b83a33ad41341149d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.085703 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '586d89c13d4d4f9b83a33ad41341149d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.086375 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '586d89c13d4d4f9b83a33ad41341149d'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.183775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1266/2529] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:40 2022] POST /identity/v3/users => generated 287 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:39:41.187801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:41.192259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:41.195907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1264/2530] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:40 2022] DELETE /identity/v3/users/586d89c13d4d4f9b83a33ad41341149d => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:39:41.205012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-289b6f0a-7c80-4cf2-9482-884637e1fad9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:41.267831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385b919b-4861-4748-bbd9-1b3a543650d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:41.268448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385b919b-4861-4748-bbd9-1b3a543650d9 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:41.268723 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385b919b-4861-4748-bbd9-1b3a543650d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:41.268990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385b919b-4861-4748-bbd9-1b3a543650d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:41.269643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385b919b-4861-4748-bbd9-1b3a543650d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:41.271289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385b919b-4861-4748-bbd9-1b3a543650d9 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:41.286528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1267/2531] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:41 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:41.288096 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:41.301437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25b0104f-0df4-4874-872b-dbd23a8a8b6b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:41.311011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67233608-86f9-4324-8974-3a18b18d11c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:41.311011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67233608-86f9-4324-8974-3a18b18d11c4 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:41.311011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67233608-86f9-4324-8974-3a18b18d11c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:41.311011 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67233608-86f9-4324-8974-3a18b18d11c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:41.311587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67233608-86f9-4324-8974-3a18b18d11c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:41.311677 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67233608-86f9-4324-8974-3a18b18d11c4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:41.338792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1265/2532] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:41 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:41.343362 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385b919b-4861-4748-bbd9-1b3a543650d9 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:41.355075 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385b919b-4861-4748-bbd9-1b3a543650d9 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:41.359954 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:41.360659 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:41.360939 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:41.361211 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] PATH_INFO: `/v3/projects/536e7476b1434683bc5a61ec98d1562d/users/af46284866c34bfbbfeed00259bc2082/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:41.378679 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] RBAC: Authorizing `identity:create_grant(project_id=536e7476b1434683bc5a61ec98d1562d, user_id=af46284866c34bfbbfeed00259bc2082, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:41.382235 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:41.460335 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1268/2533] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:41 2022] PUT /identity/v3/projects/536e7476b1434683bc5a61ec98d1562d/users/af46284866c34bfbbfeed00259bc2082/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:41.471191 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20be8e5-8ba5-46bc-ad9c-3876a1c3d705 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:41.474751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20be8e5-8ba5-46bc-ad9c-3876a1c3d705 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:41.474751 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b20be8e5-8ba5-46bc-ad9c-3876a1c3d705 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:41.501852 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b20be8e5-8ba5-46bc-ad9c-3876a1c3d705 None None] MFA Rules not processed for user `af46284866c34bfbbfeed00259bc2082`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:41.503128 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b20be8e5-8ba5-46bc-ad9c-3876a1c3d705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:41.567761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1266/2534] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:41.578483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:41.580289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:41.610562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20be8e5-8ba5-46bc-ad9c-3876a1c3d705 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:41.612220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b20be8e5-8ba5-46bc-ad9c-3876a1c3d705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:41.628443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:41.644770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bc4529-f1a5-4542-bd3b-cfcce7573910 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:41.648913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-877512e6-1628-42c6-80ad-057136b960f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:41.649612 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877512e6-1628-42c6-80ad-057136b960f8 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:41.652353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877512e6-1628-42c6-80ad-057136b960f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:41.652353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-877512e6-1628-42c6-80ad-057136b960f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:41.658834 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877512e6-1628-42c6-80ad-057136b960f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:41.660928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-877512e6-1628-42c6-80ad-057136b960f8 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:41.669837 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1269/2535] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:41 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:41.688218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b20be8e5-8ba5-46bc-ad9c-3876a1c3d705 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:41.693008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:41.693825 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:41.694304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:41.695002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] PATH_INFO: `/v3/projects/104a66c4d054477f8b3f91df7959214e` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:41.696238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] RBAC: Authorizing `identity:delete_project(project_id=104a66c4d054477f8b3f91df7959214e)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:41.698095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:41.813097 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '104a66c4d054477f8b3f91df7959214e', 'request_id': 'req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.819541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '104a66c4d054477f8b3f91df7959214e', 'request_id': 'req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.825926 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '104a66c4d054477f8b3f91df7959214e', 'request_id': 'req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.919304 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] The token cache is being invalidate because project 104a66c4d054477f8b3f91df7959214e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:41.919592 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.920238 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ff33bfcb-4dc6-4f9d-9086-8beab00cab87 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:41.921749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1267/2536] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:41 2022] DELETE /identity/v3/projects/104a66c4d054477f8b3f91df7959214e => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:41.926697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7add3920-5f87-48bb-b30a-67f9cbc08362 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:41.926997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7add3920-5f87-48bb-b30a-67f9cbc08362 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:41.927267 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7add3920-5f87-48bb-b30a-67f9cbc08362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:41.947581 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-7add3920-5f87-48bb-b30a-67f9cbc08362 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:41.953572 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-7add3920-5f87-48bb-b30a-67f9cbc08362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:42.007962 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1270/2537] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:42.015701 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53653ac4-b53b-43ad-8e8c-ca834e7834c2 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.015980 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53653ac4-b53b-43ad-8e8c-ca834e7834c2 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.016250 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53653ac4-b53b-43ad-8e8c-ca834e7834c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.036783 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-53653ac4-b53b-43ad-8e8c-ca834e7834c2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:42.037976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-53653ac4-b53b-43ad-8e8c-ca834e7834c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:42.046080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1268/2538] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:42.050328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7add3920-5f87-48bb-b30a-67f9cbc08362 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.056543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:42.060306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7add3920-5f87-48bb-b30a-67f9cbc08362 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.065006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:42.065629 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.066440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.066716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.067407 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:42.067713 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:42.069321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:42.085072 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1271/2539] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:39:42.085434 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53653ac4-b53b-43ad-8e8c-ca834e7834c2 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.094848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53653ac4-b53b-43ad-8e8c-ca834e7834c2 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.099492 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2a543f-6bc6-45e1-a77d-389090129203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:42.100080 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a543f-6bc6-45e1-a77d-389090129203 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.100353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a543f-6bc6-45e1-a77d-389090129203 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.100620 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2a543f-6bc6-45e1-a77d-389090129203 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.101237 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2a543f-6bc6-45e1-a77d-389090129203 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:42.103139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2a543f-6bc6-45e1-a77d-389090129203 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:42.189989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1269/2540] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:42 2022] POST /identity/v3/projects => generated 365 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:42.194736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.204804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c81952-c7e4-48fe-bb9d-52f7a4bebb3f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.261489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19af2ffa-5853-4813-9fe0-38cb5adcc153 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:42.262164 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19af2ffa-5853-4813-9fe0-38cb5adcc153 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.262654 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19af2ffa-5853-4813-9fe0-38cb5adcc153 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.263225 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19af2ffa-5853-4813-9fe0-38cb5adcc153 admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.272814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19af2ffa-5853-4813-9fe0-38cb5adcc153 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:42.272814 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19af2ffa-5853-4813-9fe0-38cb5adcc153 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:42.349440 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1272/2541] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:42 2022] POST /identity/v3/users => generated 283 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:42.353581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2a543f-6bc6-45e1-a77d-389090129203 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.365416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2a543f-6bc6-45e1-a77d-389090129203 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.369177 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b793f86b-0be4-42c7-a196-f1e8099004da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:42.369766 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793f86b-0be4-42c7-a196-f1e8099004da admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.370490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793f86b-0be4-42c7-a196-f1e8099004da admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.370767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b793f86b-0be4-42c7-a196-f1e8099004da admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.371562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b793f86b-0be4-42c7-a196-f1e8099004da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:42.376374 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b793f86b-0be4-42c7-a196-f1e8099004da admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:42.388146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1270/2542] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:42.389692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19af2ffa-5853-4813-9fe0-38cb5adcc153 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.402374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19af2ffa-5853-4813-9fe0-38cb5adcc153 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.411367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f7fcb36-78aa-4c16-a9fa-2c08212bac89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:42.412062 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fcb36-78aa-4c16-a9fa-2c08212bac89 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.413360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fcb36-78aa-4c16-a9fa-2c08212bac89 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.413360 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f7fcb36-78aa-4c16-a9fa-2c08212bac89 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.414183 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7fcb36-78aa-4c16-a9fa-2c08212bac89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:42.416101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f7fcb36-78aa-4c16-a9fa-2c08212bac89 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:42.430636 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1273/2543] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:42 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:42.434658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b793f86b-0be4-42c7-a196-f1e8099004da admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.446154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b793f86b-0be4-42c7-a196-f1e8099004da admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.452327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59e129c-bdfb-48d8-80a3-f41805b8a59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:42.456782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59e129c-bdfb-48d8-80a3-f41805b8a59b admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.456782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59e129c-bdfb-48d8-80a3-f41805b8a59b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.456782 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59e129c-bdfb-48d8-80a3-f41805b8a59b admin admin] PATH_INFO: `/v3/projects/8daedfd87dc0450a812804519a4b530c/users/757084f292784b49964a77f88ef5e7ca/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.471472 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59e129c-bdfb-48d8-80a3-f41805b8a59b admin admin] RBAC: Authorizing `identity:create_grant(project_id=8daedfd87dc0450a812804519a4b530c, user_id=757084f292784b49964a77f88ef5e7ca, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:42.474195 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59e129c-bdfb-48d8-80a3-f41805b8a59b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:42.548376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1271/2544] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:42 2022] PUT /identity/v3/projects/8daedfd87dc0450a812804519a4b530c/users/757084f292784b49964a77f88ef5e7ca/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:42.557172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe249f-4051-4155-a01d-fc229a4f2eb1 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.557534 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe249f-4051-4155-a01d-fc229a4f2eb1 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.557882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbe249f-4051-4155-a01d-fc229a4f2eb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.586137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-2cbe249f-4051-4155-a01d-fc229a4f2eb1 None None] MFA Rules not processed for user `757084f292784b49964a77f88ef5e7ca`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:42.587395 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2cbe249f-4051-4155-a01d-fc229a4f2eb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:42.651938 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1274/2545] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:42 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:42.656464 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59e129c-bdfb-48d8-80a3-f41805b8a59b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.698440 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:42.718872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59e129c-bdfb-48d8-80a3-f41805b8a59b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.724175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67727e00-cae3-4659-8fc5-f7006b331cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:42.724771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67727e00-cae3-4659-8fc5-f7006b331cec admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.725052 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67727e00-cae3-4659-8fc5-f7006b331cec admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.725326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67727e00-cae3-4659-8fc5-f7006b331cec admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.725960 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67727e00-cae3-4659-8fc5-f7006b331cec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:42.728029 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67727e00-cae3-4659-8fc5-f7006b331cec admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:42.826971 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1272/2546] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:42 2022] POST /identity/v3/projects => generated 363 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:42.831102 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbe249f-4051-4155-a01d-fc229a4f2eb1 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.841945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbe249f-4051-4155-a01d-fc229a4f2eb1 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.897489 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb00935-e89f-4f85-956a-2a03b121f11a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:42.898083 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb00935-e89f-4f85-956a-2a03b121f11a admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:42.898363 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb00935-e89f-4f85-956a-2a03b121f11a admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:42.898631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb00935-e89f-4f85-956a-2a03b121f11a admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:42.899311 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb00935-e89f-4f85-956a-2a03b121f11a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:42.901201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb00935-e89f-4f85-956a-2a03b121f11a admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:42.909140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:42.979816 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1275/2547] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:42 2022] POST /identity/v3/users => generated 288 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:42.983653 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67727e00-cae3-4659-8fc5-f7006b331cec admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:42.994736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67727e00-cae3-4659-8fc5-f7006b331cec admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:42.999379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aede0c07-6139-489e-b128-e847d8b4225c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:43.000002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aede0c07-6139-489e-b128-e847d8b4225c admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:43.002190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aede0c07-6139-489e-b128-e847d8b4225c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:43.002190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aede0c07-6139-489e-b128-e847d8b4225c admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:43.002190 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aede0c07-6139-489e-b128-e847d8b4225c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:43.002719 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aede0c07-6139-489e-b128-e847d8b4225c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:43.012484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1273/2548] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:42 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:39:43.016658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb00935-e89f-4f85-956a-2a03b121f11a admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:43.026482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb00935-e89f-4f85-956a-2a03b121f11a admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:43.031029 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7825ac80-ff21-43df-bebf-c9c983507690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:43.031665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7825ac80-ff21-43df-bebf-c9c983507690 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:43.031942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7825ac80-ff21-43df-bebf-c9c983507690 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:43.032214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7825ac80-ff21-43df-bebf-c9c983507690 admin admin] PATH_INFO: `/v3/projects/50ff710954d44a5eb650f579715f7225/users/d686e8ca5ef54913883911b15d6b0cad/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:43.047005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7825ac80-ff21-43df-bebf-c9c983507690 admin admin] RBAC: Authorizing `identity:create_grant(project_id=50ff710954d44a5eb650f579715f7225, user_id=d686e8ca5ef54913883911b15d6b0cad, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:43.049732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7825ac80-ff21-43df-bebf-c9c983507690 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:43.114105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1276/2549] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:43 2022] PUT /identity/v3/projects/50ff710954d44a5eb650f579715f7225/users/d686e8ca5ef54913883911b15d6b0cad/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:43.118146 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aede0c07-6139-489e-b128-e847d8b4225c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:43.132862 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aede0c07-6139-489e-b128-e847d8b4225c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:43.188508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3669ab-680d-4bfb-9a37-008cc50affbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:43.189043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3669ab-680d-4bfb-9a37-008cc50affbf admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:43.189316 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3669ab-680d-4bfb-9a37-008cc50affbf admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:43.189586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3669ab-680d-4bfb-9a37-008cc50affbf admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:43.190211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3669ab-680d-4bfb-9a37-008cc50affbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:43.191865 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3669ab-680d-4bfb-9a37-008cc50affbf admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:43.201171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1274/2550] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:43 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:43.205462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7825ac80-ff21-43df-bebf-c9c983507690 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:43.216403 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7825ac80-ff21-43df-bebf-c9c983507690 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:43.221003 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:43.221640 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:43.221922 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:43.222817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] PATH_INFO: `/v3/projects/50ff710954d44a5eb650f579715f7225/users/d686e8ca5ef54913883911b15d6b0cad/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:43.224784 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] RBAC: Authorizing `identity:create_grant(project_id=50ff710954d44a5eb650f579715f7225, user_id=d686e8ca5ef54913883911b15d6b0cad, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:43.227455 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:43.237464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1277/2551] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:43 2022] PUT /identity/v3/projects/50ff710954d44a5eb650f579715f7225/users/d686e8ca5ef54913883911b15d6b0cad/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:43.248559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85db344f-ca4d-4834-991c-11a53b48f1d0 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:43.248849 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85db344f-ca4d-4834-991c-11a53b48f1d0 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:43.249130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85db344f-ca4d-4834-991c-11a53b48f1d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:43.277063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-85db344f-ca4d-4834-991c-11a53b48f1d0 None None] MFA Rules not processed for user `d686e8ca5ef54913883911b15d6b0cad`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:43.278270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-85db344f-ca4d-4834-991c-11a53b48f1d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:43.349603 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1275/2552] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:43 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:43.361325 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:43.363054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:43.424711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e71c5b-17b8-47d0-badd-8cbb39456691 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:43.428598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:43.429197 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:43.429471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:43.429739 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:43.438918 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:43.440307 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:43.450409 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1278/2553] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:43 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:43.681354 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85db344f-ca4d-4834-991c-11a53b48f1d0 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:43.697625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85db344f-ca4d-4834-991c-11a53b48f1d0 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:43.697625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ac3921-edb0-47fb-b330-5da72e5cb57d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:43.697625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac3921-edb0-47fb-b330-5da72e5cb57d service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:43.697625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac3921-edb0-47fb-b330-5da72e5cb57d service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:43.698086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac3921-edb0-47fb-b330-5da72e5cb57d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:43.728874 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:43.757283 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ac3921-edb0-47fb-b330-5da72e5cb57d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:43.758731 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ac3921-edb0-47fb-b330-5da72e5cb57d service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:43.767691 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1276/2554] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:43 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:44.759479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:44.760969 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:44.834559 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e853bed-fcbf-489c-a002-e922c36ba96a service cinder] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:44.840260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f70d5e-2024-4392-ab8e-d369f9ad8351 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:44.840848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f70d5e-2024-4392-ab8e-d369f9ad8351 service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:44.841123 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f70d5e-2024-4392-ab8e-d369f9ad8351 service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:44.841391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f70d5e-2024-4392-ab8e-d369f9ad8351 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:44.843115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-a2f70d5e-2024-4392-ab8e-d369f9ad8351 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:44.889961 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:44.899974 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f70d5e-2024-4392-ab8e-d369f9ad8351 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:44.901567 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f70d5e-2024-4392-ab8e-d369f9ad8351 service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:44.911115 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1279/2555] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:44 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:47.269387 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ac3921-edb0-47fb-b330-5da72e5cb57d service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:47.391000 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ac3921-edb0-47fb-b330-5da72e5cb57d service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:47.396442 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:47.397084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:47.401244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:47.401244 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] PATH_INFO: `/v3/users/757084f292784b49964a77f88ef5e7ca` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:47.402298 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] RBAC: Authorizing `identity:delete_user(user_id=757084f292784b49964a77f88ef5e7ca)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:47.405161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:47.584028 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '757084f292784b49964a77f88ef5e7ca'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:47.584771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '757084f292784b49964a77f88ef5e7ca'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:47.599379 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '757084f292784b49964a77f88ef5e7ca'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:47.606697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '757084f292784b49964a77f88ef5e7ca'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:47.607450 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '757084f292784b49964a77f88ef5e7ca'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:47.678208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1277/2556] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:47 2022] DELETE /identity/v3/users/757084f292784b49964a77f88ef5e7ca => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:47.682406 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6907e0d-e8bb-4208-a9ad-b922b1069e4e None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:47.682890 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6907e0d-e8bb-4208-a9ad-b922b1069e4e None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:47.683198 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6907e0d-e8bb-4208-a9ad-b922b1069e4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:47.704284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-c6907e0d-e8bb-4208-a9ad-b922b1069e4e None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:47.705558 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c6907e0d-e8bb-4208-a9ad-b922b1069e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:47.773726 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1280/2557] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:47.783397 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:47.787535 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:47.846829 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5bb6dc8-e682-44c9-8a36-99148890cb25 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:47.850495 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4209743d-4ed9-4999-9249-7e4dbd6cd4e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:47.851129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4209743d-4ed9-4999-9249-7e4dbd6cd4e1 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:47.851451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4209743d-4ed9-4999-9249-7e4dbd6cd4e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:47.851724 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4209743d-4ed9-4999-9249-7e4dbd6cd4e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:47.857929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:47.861419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4209743d-4ed9-4999-9249-7e4dbd6cd4e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:47.863471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4209743d-4ed9-4999-9249-7e4dbd6cd4e1 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:47.873254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1278/2558] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:48.298488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6907e0d-e8bb-4208-a9ad-b922b1069e4e None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:48.300836 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-c6907e0d-e8bb-4208-a9ad-b922b1069e4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:48.323835 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6907e0d-e8bb-4208-a9ad-b922b1069e4e None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:48.336361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:48.336997 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:48.337273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:48.337543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] PATH_INFO: `/v3/projects/8daedfd87dc0450a812804519a4b530c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:48.338924 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8daedfd87dc0450a812804519a4b530c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:48.340843 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:48.445646 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8daedfd87dc0450a812804519a4b530c', 'request_id': 'req-80ca3707-041f-4258-a328-8391c5b28f78'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.502541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8daedfd87dc0450a812804519a4b530c', 'request_id': 'req-80ca3707-041f-4258-a328-8391c5b28f78'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.509010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8daedfd87dc0450a812804519a4b530c', 'request_id': 'req-80ca3707-041f-4258-a328-8391c5b28f78'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.522244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] The token cache is being invalidate because project 8daedfd87dc0450a812804519a4b530c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:48.522533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.523219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.526584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1281/2559] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:48 2022] DELETE /identity/v3/projects/8daedfd87dc0450a812804519a4b530c => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:48.530648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4209743d-4ed9-4999-9249-7e4dbd6cd4e1 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:48.541944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4209743d-4ed9-4999-9249-7e4dbd6cd4e1 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:48.639831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:48.640708 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:48.641023 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:48.641437 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] PATH_INFO: `/v3/users/d686e8ca5ef54913883911b15d6b0cad` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:48.642805 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d686e8ca5ef54913883911b15d6b0cad)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:48.645524 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:48.865417 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd686e8ca5ef54913883911b15d6b0cad'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.866077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd686e8ca5ef54913883911b15d6b0cad'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.881604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd686e8ca5ef54913883911b15d6b0cad'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.890046 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd686e8ca5ef54913883911b15d6b0cad'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.890705 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d94da644-946a-4853-a4d5-0ac7dcdfe407 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd686e8ca5ef54913883911b15d6b0cad'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:48.974792 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1279/2560] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:48 2022] DELETE /identity/v3/users/d686e8ca5ef54913883911b15d6b0cad => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:49.457063 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:49.458369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:49.473352 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ca3707-041f-4258-a328-8391c5b28f78 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:49.530796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:49.531478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:49.531754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:49.532028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] PATH_INFO: `/v3/projects/50ff710954d44a5eb650f579715f7225` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:49.533134 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] RBAC: Authorizing `identity:delete_project(project_id=50ff710954d44a5eb650f579715f7225)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:49.535732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:49.552618 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:49.654038 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '50ff710954d44a5eb650f579715f7225', 'request_id': 'req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:49.751220 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '50ff710954d44a5eb650f579715f7225', 'request_id': 'req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:49.757488 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '50ff710954d44a5eb650f579715f7225', 'request_id': 'req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:49.770564 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] The token cache is being invalidate because project 50ff710954d44a5eb650f579715f7225 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:49.770861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:49.771541 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-70d32f3f-0d86-4378-b8a5-f04cbe38c03b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:49.773306 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1282/2561] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:49 2022] DELETE /identity/v3/projects/50ff710954d44a5eb650f579715f7225 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:49.778478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da356b0-b692-4e3d-b44c-58a6a7981ac3 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:49.784729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da356b0-b692-4e3d-b44c-58a6a7981ac3 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:49.784729 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da356b0-b692-4e3d-b44c-58a6a7981ac3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:49.807063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-0da356b0-b692-4e3d-b44c-58a6a7981ac3 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:49.807063 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-0da356b0-b692-4e3d-b44c-58a6a7981ac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:49.869159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1280/2562] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:49.877017 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1328ac5-e844-43bf-9b00-bf35669ba724 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:49.877372 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1328ac5-e844-43bf-9b00-bf35669ba724 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:49.877716 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1328ac5-e844-43bf-9b00-bf35669ba724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:49.898772 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-f1328ac5-e844-43bf-9b00-bf35669ba724 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:49.900147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-f1328ac5-e844-43bf-9b00-bf35669ba724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:49.908201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1283/2563] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:49.913077 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da356b0-b692-4e3d-b44c-58a6a7981ac3 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:49.922907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da356b0-b692-4e3d-b44c-58a6a7981ac3 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:49.927559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:49.928147 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:49.928425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:49.928699 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:49.929343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:49.929644 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:49.931386 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:49.940276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:49.943421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1281/2564] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:39:49.950009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1328ac5-e844-43bf-9b00-bf35669ba724 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:49.960219 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1328ac5-e844-43bf-9b00-bf35669ba724 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:49.965189 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032261df-3f07-4cd5-b8b5-6e4e7796822f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:49.965778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032261df-3f07-4cd5-b8b5-6e4e7796822f admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:49.966054 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032261df-3f07-4cd5-b8b5-6e4e7796822f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:49.966326 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032261df-3f07-4cd5-b8b5-6e4e7796822f admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:49.967018 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032261df-3f07-4cd5-b8b5-6e4e7796822f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:49.968914 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032261df-3f07-4cd5-b8b5-6e4e7796822f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.055199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1284/2565] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:49 2022] POST /identity/v3/projects => generated 379 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:50.059487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.069536 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e69ea7c-41fe-4b2c-bb6c-de21c01587e7 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:50.125468 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6cdc44-17cf-40dd-b26d-6cc656454c43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:50.126067 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6cdc44-17cf-40dd-b26d-6cc656454c43 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.126348 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6cdc44-17cf-40dd-b26d-6cc656454c43 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.126620 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6cdc44-17cf-40dd-b26d-6cc656454c43 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.127274 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6cdc44-17cf-40dd-b26d-6cc656454c43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:50.129219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df6cdc44-17cf-40dd-b26d-6cc656454c43 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.214388 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1282/2566] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:50 2022] POST /identity/v3/users => generated 290 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:50.218673 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032261df-3f07-4cd5-b8b5-6e4e7796822f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.228967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032261df-3f07-4cd5-b8b5-6e4e7796822f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:50.233620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1494616-dac2-4448-8dfe-5f0922add5ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:50.234207 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1494616-dac2-4448-8dfe-5f0922add5ec admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.234593 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1494616-dac2-4448-8dfe-5f0922add5ec admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.234874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1494616-dac2-4448-8dfe-5f0922add5ec admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.235566 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1494616-dac2-4448-8dfe-5f0922add5ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:50.237141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1494616-dac2-4448-8dfe-5f0922add5ec admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.247033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1285/2567] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:50 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:50.250847 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6cdc44-17cf-40dd-b26d-6cc656454c43 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.261733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6cdc44-17cf-40dd-b26d-6cc656454c43 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:50.266587 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8a9612-3ef2-4075-90ea-26c81f85ca0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:50.267218 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8a9612-3ef2-4075-90ea-26c81f85ca0f admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.267503 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8a9612-3ef2-4075-90ea-26c81f85ca0f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.267773 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8a9612-3ef2-4075-90ea-26c81f85ca0f admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.268415 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8a9612-3ef2-4075-90ea-26c81f85ca0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:50.270640 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8a9612-3ef2-4075-90ea-26c81f85ca0f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.281422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1283/2568] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:50.285607 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1494616-dac2-4448-8dfe-5f0922add5ec admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.295848 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1494616-dac2-4448-8dfe-5f0922add5ec admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:50.300482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670dbe00-7b8e-4b58-950e-8d5e309b0584 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:50.301121 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670dbe00-7b8e-4b58-950e-8d5e309b0584 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.301399 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670dbe00-7b8e-4b58-950e-8d5e309b0584 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.301692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670dbe00-7b8e-4b58-950e-8d5e309b0584 admin admin] PATH_INFO: `/v3/projects/e2feca3664b64fb180407fd69190bdae/users/20cd1984253e4be3a4e14a26e7d6647c/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.316754 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670dbe00-7b8e-4b58-950e-8d5e309b0584 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e2feca3664b64fb180407fd69190bdae, user_id=20cd1984253e4be3a4e14a26e7d6647c, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:50.319415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-670dbe00-7b8e-4b58-950e-8d5e309b0584 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.385433 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1286/2569] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:50 2022] PUT /identity/v3/projects/e2feca3664b64fb180407fd69190bdae/users/20cd1984253e4be3a4e14a26e7d6647c/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:50.397487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf9504-96fb-4f2e-8b84-4e29a92e21d7 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.397777 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf9504-96fb-4f2e-8b84-4e29a92e21d7 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.398056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bf9504-96fb-4f2e-8b84-4e29a92e21d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.425213 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-c7bf9504-96fb-4f2e-8b84-4e29a92e21d7 None None] MFA Rules not processed for user `20cd1984253e4be3a4e14a26e7d6647c`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:50.426423 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-c7bf9504-96fb-4f2e-8b84-4e29a92e21d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:50.491686 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1284/2570] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:50 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:50.495921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670dbe00-7b8e-4b58-950e-8d5e309b0584 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.558500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670dbe00-7b8e-4b58-950e-8d5e309b0584 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:50.563173 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51e5bf2-44db-4f60-bd6a-b7574e756467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:50.563775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51e5bf2-44db-4f60-bd6a-b7574e756467 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.564056 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51e5bf2-44db-4f60-bd6a-b7574e756467 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.564328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51e5bf2-44db-4f60-bd6a-b7574e756467 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.564959 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51e5bf2-44db-4f60-bd6a-b7574e756467 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:50.567622 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51e5bf2-44db-4f60-bd6a-b7574e756467 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.659921 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1287/2571] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:50 2022] POST /identity/v3/projects => generated 379 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:50.664056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bf9504-96fb-4f2e-8b84-4e29a92e21d7 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.675518 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bf9504-96fb-4f2e-8b84-4e29a92e21d7 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:50.740612 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de658fc1-fc6d-413b-b708-f4a5b89e4c74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:50.741262 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de658fc1-fc6d-413b-b708-f4a5b89e4c74 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.741499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de658fc1-fc6d-413b-b708-f4a5b89e4c74 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.741774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de658fc1-fc6d-413b-b708-f4a5b89e4c74 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.742401 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de658fc1-fc6d-413b-b708-f4a5b89e4c74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:50.744384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de658fc1-fc6d-413b-b708-f4a5b89e4c74 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.814222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:50.821896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1285/2572] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:50 2022] POST /identity/v3/users => generated 296 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:50.825328 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51e5bf2-44db-4f60-bd6a-b7574e756467 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.836006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51e5bf2-44db-4f60-bd6a-b7574e756467 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:50.841110 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8086ed28-e400-43f4-ab53-a92f35d11bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:50.847273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8086ed28-e400-43f4-ab53-a92f35d11bfc admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.847273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8086ed28-e400-43f4-ab53-a92f35d11bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.847273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8086ed28-e400-43f4-ab53-a92f35d11bfc admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.847273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8086ed28-e400-43f4-ab53-a92f35d11bfc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:50.847273 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8086ed28-e400-43f4-ab53-a92f35d11bfc admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.855266 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1288/2573] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:50.859384 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de658fc1-fc6d-413b-b708-f4a5b89e4c74 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.870144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de658fc1-fc6d-413b-b708-f4a5b89e4c74 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:50.875419 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ca4bb0-34e1-44a6-99fa-6c7dc1b8b6c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:50.875651 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca4bb0-34e1-44a6-99fa-6c7dc1b8b6c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:50.875932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca4bb0-34e1-44a6-99fa-6c7dc1b8b6c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:50.876231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca4bb0-34e1-44a6-99fa-6c7dc1b8b6c2 admin admin] PATH_INFO: `/v3/projects/942a8b5944354e35af5c0717812c1532/users/1bfa86a573cf41d68d78185c09bc8f5f/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:50.891418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ca4bb0-34e1-44a6-99fa-6c7dc1b8b6c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=942a8b5944354e35af5c0717812c1532, user_id=1bfa86a573cf41d68d78185c09bc8f5f, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:50.894203 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ca4bb0-34e1-44a6-99fa-6c7dc1b8b6c2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:50.976116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1286/2574] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:50 2022] PUT /identity/v3/projects/942a8b5944354e35af5c0717812c1532/users/1bfa86a573cf41d68d78185c09bc8f5f/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:50.980073 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8086ed28-e400-43f4-ab53-a92f35d11bfc admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:50.991425 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8086ed28-e400-43f4-ab53-a92f35d11bfc admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:51.003752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:51.056091 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b332065e-8e02-43f3-999f-29374709cfad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:51.056699 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b332065e-8e02-43f3-999f-29374709cfad admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:51.057001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b332065e-8e02-43f3-999f-29374709cfad admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:51.057269 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b332065e-8e02-43f3-999f-29374709cfad admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:51.057913 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b332065e-8e02-43f3-999f-29374709cfad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:51.059573 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b332065e-8e02-43f3-999f-29374709cfad admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:51.069272 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1289/2575] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:50 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:51.073761 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ca4bb0-34e1-44a6-99fa-6c7dc1b8b6c2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:51.084158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ca4bb0-34e1-44a6-99fa-6c7dc1b8b6c2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:51.089009 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:51.089648 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:51.091220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:51.091220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] PATH_INFO: `/v3/projects/942a8b5944354e35af5c0717812c1532/users/1bfa86a573cf41d68d78185c09bc8f5f/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:51.092114 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] RBAC: Authorizing `identity:create_grant(project_id=942a8b5944354e35af5c0717812c1532, user_id=1bfa86a573cf41d68d78185c09bc8f5f, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:51.094710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:51.105183 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1287/2576] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:51 2022] PUT /identity/v3/projects/942a8b5944354e35af5c0717812c1532/users/1bfa86a573cf41d68d78185c09bc8f5f/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:51.115953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69971373-895b-491f-a213-4fca30e0163c None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:51.116256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69971373-895b-491f-a213-4fca30e0163c None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:51.116531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69971373-895b-491f-a213-4fca30e0163c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:51.144533 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-69971373-895b-491f-a213-4fca30e0163c None None] MFA Rules not processed for user `1bfa86a573cf41d68d78185c09bc8f5f`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:51.145130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-69971373-895b-491f-a213-4fca30e0163c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:51.213655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1290/2577] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:51 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:51.225148 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:51.226903 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:51.282870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787de1b1-0f11-409d-9f3a-4cdd1263318a admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:51.287480 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0925efa5-6acd-4764-ba74-87c04578520e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:51.288081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0925efa5-6acd-4764-ba74-87c04578520e service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:51.288358 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0925efa5-6acd-4764-ba74-87c04578520e service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:51.288629 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0925efa5-6acd-4764-ba74-87c04578520e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:51.297759 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0925efa5-6acd-4764-ba74-87c04578520e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:51.299208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0925efa5-6acd-4764-ba74-87c04578520e service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:51.309102 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1288/2578] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:51 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:51.808260 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69971373-895b-491f-a213-4fca30e0163c None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:51.869316 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69971373-895b-491f-a213-4fca30e0163c None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:51.873967 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:51.874584 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:51.874859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:51.875178 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] PATH_INFO: `/v3/users/20cd1984253e4be3a4e14a26e7d6647c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:51.876292 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] RBAC: Authorizing `identity:delete_user(user_id=20cd1984253e4be3a4e14a26e7d6647c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:51.878766 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:52.036538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:52.042449 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20cd1984253e4be3a4e14a26e7d6647c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:52.091385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20cd1984253e4be3a4e14a26e7d6647c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:52.103697 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20cd1984253e4be3a4e14a26e7d6647c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:52.110543 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20cd1984253e4be3a4e14a26e7d6647c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:52.111240 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20cd1984253e4be3a4e14a26e7d6647c'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:52.117789 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1291/2579] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:51 2022] DELETE /identity/v3/users/20cd1984253e4be3a4e14a26e7d6647c => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:52.122199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02864bd-8e83-45a0-b357-71f50de19443 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:52.122687 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02864bd-8e83-45a0-b357-71f50de19443 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:52.122888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02864bd-8e83-45a0-b357-71f50de19443 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:52.142124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-a02864bd-8e83-45a0-b357-71f50de19443 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:52.143350 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a02864bd-8e83-45a0-b357-71f50de19443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:52.208788 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1289/2580] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:52.215833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:52.217626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:52.276369 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d608284a-f04f-49cd-af82-b660c2cf066f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:52.280209 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:52.281015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:52.284796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:52.284796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:52.291415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:52.292861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:52.303177 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1292/2581] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:52.906226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02864bd-8e83-45a0-b357-71f50de19443 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:52.907696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a02864bd-8e83-45a0-b357-71f50de19443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:52.920151 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02864bd-8e83-45a0-b357-71f50de19443 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:52.924937 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:52.925577 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:52.925855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:52.926129 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] PATH_INFO: `/v3/projects/e2feca3664b64fb180407fd69190bdae` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:52.927441 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e2feca3664b64fb180407fd69190bdae)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:52.929945 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:53.020220 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e2feca3664b64fb180407fd69190bdae', 'request_id': 'req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.026425 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e2feca3664b64fb180407fd69190bdae', 'request_id': 'req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.030655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:53.032884 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e2feca3664b64fb180407fd69190bdae', 'request_id': 'req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.050598 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:53.052052 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:53.066740 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c04a14-cb34-4015-92a7-d615149eb540 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:53.101507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] The token cache is being invalidate because project e2feca3664b64fb180407fd69190bdae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:53.101895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.102642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.107158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1290/2582] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:52 2022] DELETE /identity/v3/projects/e2feca3664b64fb180407fd69190bdae => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:53.111845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:53.113602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:53.141026 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:53.141722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:53.142468 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:53.142752 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] PATH_INFO: `/v3/users/af46284866c34bfbbfeed00259bc2082` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:53.145141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] RBAC: Authorizing `identity:delete_user(user_id=af46284866c34bfbbfeed00259bc2082)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:53.147130 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:53.182482 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00d8ee4-bed7-46ef-a22f-e4de2f52118e admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:53.187462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:53.188216 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:53.188499 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:53.188774 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] PATH_INFO: `/v3/users/1bfa86a573cf41d68d78185c09bc8f5f` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:53.190095 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1bfa86a573cf41d68d78185c09bc8f5f)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:53.192607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:53.319188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af46284866c34bfbbfeed00259bc2082'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.356883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1bfa86a573cf41d68d78185c09bc8f5f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.357552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1bfa86a573cf41d68d78185c09bc8f5f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.369784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1bfa86a573cf41d68d78185c09bc8f5f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.376843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1bfa86a573cf41d68d78185c09bc8f5f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.377584 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1bfa86a573cf41d68d78185c09bc8f5f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.416711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af46284866c34bfbbfeed00259bc2082'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.423626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:53.430660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af46284866c34bfbbfeed00259bc2082'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.437749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af46284866c34bfbbfeed00259bc2082'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.438597 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1ed66b24-5c6a-41c7-9f42-cdf870ca8f76 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af46284866c34bfbbfeed00259bc2082'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:53.445193 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1293/2583] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:53 2022] DELETE /identity/v3/users/af46284866c34bfbbfeed00259bc2082 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:53.449809 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d1bccb-79ba-4060-947e-108ad3fe1dfc None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:53.450096 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d1bccb-79ba-4060-947e-108ad3fe1dfc None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:53.450378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d1bccb-79ba-4060-947e-108ad3fe1dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:53.470885 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-66d1bccb-79ba-4060-947e-108ad3fe1dfc None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:53.471228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-66d1bccb-79ba-4060-947e-108ad3fe1dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:53.485553 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1291/2584] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:53 2022] DELETE /identity/v3/users/1bfa86a573cf41d68d78185c09bc8f5f => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:53.531588 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1294/2585] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:53.543429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:53.545015 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:53.603284 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f134a53-594c-4afa-a7f2-fe607c0db370 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:53.607038 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:53.607657 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:53.607928 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:53.608199 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:53.628621 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:53.677596 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:53.679282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:53.689550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1292/2586] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:53.977815 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d1bccb-79ba-4060-947e-108ad3fe1dfc None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:53.980665 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-66d1bccb-79ba-4060-947e-108ad3fe1dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:53.999095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d1bccb-79ba-4060-947e-108ad3fe1dfc None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:54.005086 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:54.006055 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.006434 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.007105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] PATH_INFO: `/v3/projects/942a8b5944354e35af5c0717812c1532` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.008261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] RBAC: Authorizing `identity:delete_project(project_id=942a8b5944354e35af5c0717812c1532)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:54.022222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:54.154980 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '942a8b5944354e35af5c0717812c1532', 'request_id': 'req-aa6fcad8-129a-48ad-87bc-01f27d56800e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.224550 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:54.226083 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:54.246265 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '942a8b5944354e35af5c0717812c1532', 'request_id': 'req-aa6fcad8-129a-48ad-87bc-01f27d56800e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.251136 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8461a0-db21-4a27-a21e-fda7393fc802 service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:54.261139 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '942a8b5944354e35af5c0717812c1532', 'request_id': 'req-aa6fcad8-129a-48ad-87bc-01f27d56800e'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.275743 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] The token cache is being invalidate because project 942a8b5944354e35af5c0717812c1532 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:54.285246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.285246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-aa6fcad8-129a-48ad-87bc-01f27d56800e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.285246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1295/2587] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:53 2022] DELETE /identity/v3/projects/942a8b5944354e35af5c0717812c1532 => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:39:54.285246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e026bd7-a671-4af3-9ddf-352cc2d81554 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.285246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e026bd7-a671-4af3-9ddf-352cc2d81554 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.285246 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e026bd7-a671-4af3-9ddf-352cc2d81554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.310389 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-9e026bd7-a671-4af3-9ddf-352cc2d81554 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:54.311644 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-9e026bd7-a671-4af3-9ddf-352cc2d81554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:54.323487 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:54.324226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.324508 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.324784 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] PATH_INFO: `/v3/projects/536e7476b1434683bc5a61ec98d1562d` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.325920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] RBAC: Authorizing `identity:delete_project(project_id=536e7476b1434683bc5a61ec98d1562d)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:54.329138 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:54.381827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1296/2588] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:54.389008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ed183-42a2-400c-a85a-2692fb8dba85 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.389294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ed183-42a2-400c-a85a-2692fb8dba85 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.389633 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933ed183-42a2-400c-a85a-2692fb8dba85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.398462 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:54.411746 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-933ed183-42a2-400c-a85a-2692fb8dba85 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:54.412990 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-933ed183-42a2-400c-a85a-2692fb8dba85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:54.421259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1297/2589] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:54.426113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933ed183-42a2-400c-a85a-2692fb8dba85 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:54.433924 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '536e7476b1434683bc5a61ec98d1562d', 'request_id': 'req-a834c476-eff7-45f0-a89e-98f73333eafe'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.435706 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933ed183-42a2-400c-a85a-2692fb8dba85 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:54.441312 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '536e7476b1434683bc5a61ec98d1562d', 'request_id': 'req-a834c476-eff7-45f0-a89e-98f73333eafe'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.447864 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '536e7476b1434683bc5a61ec98d1562d', 'request_id': 'req-a834c476-eff7-45f0-a89e-98f73333eafe'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.507141 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:54.507763 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.508046 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.508366 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.509028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:54.509340 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:54.510964 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:54.531101 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1298/2590] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 19 22:39:54.531464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:54.540319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] The token cache is being invalidate because project 536e7476b1434683bc5a61ec98d1562d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:54.540319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.540979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:54.542988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1293/2591] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:54 2022] DELETE /identity/v3/projects/536e7476b1434683bc5a61ec98d1562d => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:54.557963 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7b9a12-16a8-437f-9b73-44d603626372 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:54.570735 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5eb698-86ec-4213-b72e-a438aa16bc31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:54.571436 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5eb698-86ec-4213-b72e-a438aa16bc31 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.571701 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5eb698-86ec-4213-b72e-a438aa16bc31 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.571976 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5eb698-86ec-4213-b72e-a438aa16bc31 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.573506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5eb698-86ec-4213-b72e-a438aa16bc31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:54.575523 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5eb698-86ec-4213-b72e-a438aa16bc31 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:54.715190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1299/2592] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:54 2022] POST /identity/v3/projects => generated 381 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:39:54.716831 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:54.718542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:54.739478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a834c476-eff7-45f0-a89e-98f73333eafe admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:54.824834 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:54.839747 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ed5156-dc0a-4dcf-aefa-98cd38df885b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:54.840367 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ed5156-dc0a-4dcf-aefa-98cd38df885b admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.840638 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ed5156-dc0a-4dcf-aefa-98cd38df885b admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.842002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ed5156-dc0a-4dcf-aefa-98cd38df885b admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.842002 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ed5156-dc0a-4dcf-aefa-98cd38df885b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:54.843562 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ed5156-dc0a-4dcf-aefa-98cd38df885b admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:54.929124 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1294/2593] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:54 2022] POST /identity/v3/users => generated 291 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:54.933631 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5eb698-86ec-4213-b72e-a438aa16bc31 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:54.945035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5eb698-86ec-4213-b72e-a438aa16bc31 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:54.949786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89d001f-d1fc-4bd3-afb3-a860ef151270 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:54.950398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89d001f-d1fc-4bd3-afb3-a860ef151270 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.950684 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89d001f-d1fc-4bd3-afb3-a860ef151270 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.950978 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89d001f-d1fc-4bd3-afb3-a860ef151270 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.951643 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89d001f-d1fc-4bd3-afb3-a860ef151270 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:54.953261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89d001f-d1fc-4bd3-afb3-a860ef151270 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:54.964432 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1300/2594] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 19 22:39:54.968489 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ed5156-dc0a-4dcf-aefa-98cd38df885b admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:54.979424 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ed5156-dc0a-4dcf-aefa-98cd38df885b admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:54.984532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80d1ccd-0f26-47c6-8a09-ce3f6dbd5e8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:54.985206 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d1ccd-0f26-47c6-8a09-ce3f6dbd5e8a admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:54.985488 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d1ccd-0f26-47c6-8a09-ce3f6dbd5e8a admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:54.985786 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80d1ccd-0f26-47c6-8a09-ce3f6dbd5e8a admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:54.986459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80d1ccd-0f26-47c6-8a09-ce3f6dbd5e8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:54.988188 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80d1ccd-0f26-47c6-8a09-ce3f6dbd5e8a admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:55.002113 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1295/2595] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:55.004012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89d001f-d1fc-4bd3-afb3-a860ef151270 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:55.015827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89d001f-d1fc-4bd3-afb3-a860ef151270 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:55.021433 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa5f761-5a1c-4760-b713-43bd85751f24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:55.022107 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa5f761-5a1c-4760-b713-43bd85751f24 admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.022390 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa5f761-5a1c-4760-b713-43bd85751f24 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.022664 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa5f761-5a1c-4760-b713-43bd85751f24 admin admin] PATH_INFO: `/v3/projects/19e2ecb6ee714037a23b0b5dff4af00a/users/12f083187d794942b6422375506d9707/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.038875 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa5f761-5a1c-4760-b713-43bd85751f24 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19e2ecb6ee714037a23b0b5dff4af00a, user_id=12f083187d794942b6422375506d9707, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:55.041561 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa5f761-5a1c-4760-b713-43bd85751f24 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:55.091945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1301/2596] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:55 2022] PUT /identity/v3/projects/19e2ecb6ee714037a23b0b5dff4af00a/users/12f083187d794942b6422375506d9707/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:39:55.103344 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3ab0e-4f6f-4125-981e-44a5c0967d1e None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.103963 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3ab0e-4f6f-4125-981e-44a5c0967d1e None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.104610 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3ab0e-4f6f-4125-981e-44a5c0967d1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.140681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-e9f3ab0e-4f6f-4125-981e-44a5c0967d1e None None] MFA Rules not processed for user `12f083187d794942b6422375506d9707`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:55.142343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-e9f3ab0e-4f6f-4125-981e-44a5c0967d1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:55.210688 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1296/2597] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:55 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:55.215531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa5f761-5a1c-4760-b713-43bd85751f24 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:55.280718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa5f761-5a1c-4760-b713-43bd85751f24 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:55.286098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febdd2cc-a417-419b-919a-a15077bdd177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:55.286704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febdd2cc-a417-419b-919a-a15077bdd177 admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.287008 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febdd2cc-a417-419b-919a-a15077bdd177 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.287284 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febdd2cc-a417-419b-919a-a15077bdd177 admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.287930 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febdd2cc-a417-419b-919a-a15077bdd177 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:55.289863 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febdd2cc-a417-419b-919a-a15077bdd177 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:55.367851 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1302/2598] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:55 2022] POST /identity/v3/projects => generated 383 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:55.371999 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f3ab0e-4f6f-4125-981e-44a5c0967d1e None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:55.383956 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f3ab0e-4f6f-4125-981e-44a5c0967d1e None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:55.446414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0426911a-7ba7-4ad3-a467-06fd75b64a4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:55.447047 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0426911a-7ba7-4ad3-a467-06fd75b64a4f admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.447326 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0426911a-7ba7-4ad3-a467-06fd75b64a4f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.447597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0426911a-7ba7-4ad3-a467-06fd75b64a4f admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.448221 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0426911a-7ba7-4ad3-a467-06fd75b64a4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:55.450145 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0426911a-7ba7-4ad3-a467-06fd75b64a4f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:55.548796 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1297/2599] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:55 2022] POST /identity/v3/users => generated 298 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:55.552500 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febdd2cc-a417-419b-919a-a15077bdd177 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:55.563166 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febdd2cc-a417-419b-919a-a15077bdd177 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:55.568384 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b4cab8-3ccd-45be-887d-fef7046f3025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:55.569016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b4cab8-3ccd-45be-887d-fef7046f3025 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.569319 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b4cab8-3ccd-45be-887d-fef7046f3025 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.569600 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94b4cab8-3ccd-45be-887d-fef7046f3025 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.570244 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b4cab8-3ccd-45be-887d-fef7046f3025 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:55.571862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94b4cab8-3ccd-45be-887d-fef7046f3025 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:55.584398 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1303/2600] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:55 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:55.588859 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0426911a-7ba7-4ad3-a467-06fd75b64a4f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:55.603578 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0426911a-7ba7-4ad3-a467-06fd75b64a4f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:55.608615 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4574af55-4a46-4f00-96a8-a0ab2382f80d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:55.609286 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4574af55-4a46-4f00-96a8-a0ab2382f80d admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.609569 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4574af55-4a46-4f00-96a8-a0ab2382f80d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.609857 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4574af55-4a46-4f00-96a8-a0ab2382f80d admin admin] PATH_INFO: `/v3/projects/5f728147c0f743ab980508bdd0aa4797/users/53df6be4d82048819ab0cab5dd6b7947/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.630279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4574af55-4a46-4f00-96a8-a0ab2382f80d admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f728147c0f743ab980508bdd0aa4797, user_id=53df6be4d82048819ab0cab5dd6b7947, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:55.633039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4574af55-4a46-4f00-96a8-a0ab2382f80d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:55.698912 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1298/2601] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:55 2022] PUT /identity/v3/projects/5f728147c0f743ab980508bdd0aa4797/users/53df6be4d82048819ab0cab5dd6b7947/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:55.703040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b4cab8-3ccd-45be-887d-fef7046f3025 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:55.714074 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94b4cab8-3ccd-45be-887d-fef7046f3025 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:55.717761 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:55.774020 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd8e4af-ac7e-4312-b1b5-3994ced36b99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:55.774621 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8e4af-ac7e-4312-b1b5-3994ced36b99 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.774898 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8e4af-ac7e-4312-b1b5-3994ced36b99 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.775199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd8e4af-ac7e-4312-b1b5-3994ced36b99 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.775847 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd8e4af-ac7e-4312-b1b5-3994ced36b99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:55.777528 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd8e4af-ac7e-4312-b1b5-3994ced36b99 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:55.787827 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1304/2602] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:55 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:55.795143 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4574af55-4a46-4f00-96a8-a0ab2382f80d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:55.809227 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4574af55-4a46-4f00-96a8-a0ab2382f80d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:55.813832 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:55.814581 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.814855 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.815154 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] PATH_INFO: `/v3/projects/5f728147c0f743ab980508bdd0aa4797/users/53df6be4d82048819ab0cab5dd6b7947/roles/5573fc131a7b421585c72c4a56d84f4b` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.818507 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f728147c0f743ab980508bdd0aa4797, user_id=53df6be4d82048819ab0cab5dd6b7947, role_id=5573fc131a7b421585c72c4a56d84f4b)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:55.821248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:55.833843 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1299/2603] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:55 2022] PUT /identity/v3/projects/5f728147c0f743ab980508bdd0aa4797/users/53df6be4d82048819ab0cab5dd6b7947/roles/5573fc131a7b421585c72c4a56d84f4b => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:55.844431 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141825d2-2c75-4b74-bae8-6718295bbc57 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:55.844714 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141825d2-2c75-4b74-bae8-6718295bbc57 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:55.844986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141825d2-2c75-4b74-bae8-6718295bbc57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:55.879770 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-141825d2-2c75-4b74-bae8-6718295bbc57 None None] MFA Rules not processed for user `53df6be4d82048819ab0cab5dd6b7947`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:55.881015 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-141825d2-2c75-4b74-bae8-6718295bbc57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:55.955217 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1305/2604] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:55 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:55.966722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:55.969428 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:55.980389 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:56.028339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba0815c-744f-4155-8762-b40c5d9c5219 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:56.032008 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927204f-cec0-4c58-9f84-d259ea3054e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:56.032607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927204f-cec0-4c58-9f84-d259ea3054e0 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:56.032885 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927204f-cec0-4c58-9f84-d259ea3054e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:56.033158 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d927204f-cec0-4c58-9f84-d259ea3054e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:56.042103 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d927204f-cec0-4c58-9f84-d259ea3054e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:56.043571 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d927204f-cec0-4c58-9f84-d259ea3054e0 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:56.054763 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1300/2605] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:55 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:56.199040 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141825d2-2c75-4b74-bae8-6718295bbc57 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:56.208923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141825d2-2c75-4b74-bae8-6718295bbc57 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:56.213151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-debe43b1-12e7-4a4f-8cdb-6537f2094a55 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:56.214786 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe43b1-12e7-4a4f-8cdb-6537f2094a55 service cinder] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:56.215111 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe43b1-12e7-4a4f-8cdb-6537f2094a55 service cinder] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:56.215392 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-debe43b1-12e7-4a4f-8cdb-6537f2094a55 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:56.276249 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debe43b1-12e7-4a4f-8cdb-6537f2094a55 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:56.277693 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-debe43b1-12e7-4a4f-8cdb-6537f2094a55 service cinder] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:56.288343 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1306/2606] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:56 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:56.750636 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927204f-cec0-4c58-9f84-d259ea3054e0 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:56.812742 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d927204f-cec0-4c58-9f84-d259ea3054e0 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:56.817265 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:56.817878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:56.818152 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:56.818429 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] PATH_INFO: `/v3/users/12f083187d794942b6422375506d9707` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:56.819538 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=12f083187d794942b6422375506d9707)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:56.821418 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:56.974421 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12f083187d794942b6422375506d9707'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:56.975091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12f083187d794942b6422375506d9707'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:56.986602 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12f083187d794942b6422375506d9707'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:56.993363 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12f083187d794942b6422375506d9707'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:56.993989 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12f083187d794942b6422375506d9707'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:57.041130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1301/2607] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:56 2022] DELETE /identity/v3/users/12f083187d794942b6422375506d9707 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:57.045507 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf50a4f-858e-46d0-8c4d-b1d72f7c26c6 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:57.045796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf50a4f-858e-46d0-8c4d-b1d72f7c26c6 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:57.046068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf50a4f-858e-46d0-8c4d-b1d72f7c26c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:57.066080 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-cbf50a4f-858e-46d0-8c4d-b1d72f7c26c6 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:57.067318 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-cbf50a4f-858e-46d0-8c4d-b1d72f7c26c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:57.087282 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=100 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:57.124445 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1307/2608] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:57.134559 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:57.136069 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:57.168034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:57.192342 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e54b2e-6cbf-4547-b20b-5bc132599b55 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:57.196020 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446ed79-0b8b-448e-8f49-aaf711828eee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:57.196609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446ed79-0b8b-448e-8f49-aaf711828eee service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:57.196888 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446ed79-0b8b-448e-8f49-aaf711828eee service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:57.197161 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3446ed79-0b8b-448e-8f49-aaf711828eee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:57.207144 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3446ed79-0b8b-448e-8f49-aaf711828eee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:57.208583 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3446ed79-0b8b-448e-8f49-aaf711828eee service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:57.218230 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1302/2609] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:39:57.598030 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf50a4f-858e-46d0-8c4d-b1d72f7c26c6 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:57.599515 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-cbf50a4f-858e-46d0-8c4d-b1d72f7c26c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:57.611799 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf50a4f-858e-46d0-8c4d-b1d72f7c26c6 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:57.617025 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:57.617658 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:57.617935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:57.618204 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] PATH_INFO: `/v3/projects/19e2ecb6ee714037a23b0b5dff4af00a` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:57.619365 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19e2ecb6ee714037a23b0b5dff4af00a)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:57.623905 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:57.722800 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19e2ecb6ee714037a23b0b5dff4af00a', 'request_id': 'req-94864f0d-e5b6-4bac-8406-604a7764b763'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:57.793557 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19e2ecb6ee714037a23b0b5dff4af00a', 'request_id': 'req-94864f0d-e5b6-4bac-8406-604a7764b763'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:57.800310 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19e2ecb6ee714037a23b0b5dff4af00a', 'request_id': 'req-94864f0d-e5b6-4bac-8406-604a7764b763'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:57.811539 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] The token cache is being invalidate because project 19e2ecb6ee714037a23b0b5dff4af00a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:57.811832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:57.812460 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:57.813962 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1308/2610] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:57 2022] DELETE /identity/v3/projects/19e2ecb6ee714037a23b0b5dff4af00a => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:57.818064 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446ed79-0b8b-448e-8f49-aaf711828eee service neutron] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:57.827597 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3446ed79-0b8b-448e-8f49-aaf711828eee service neutron] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:57.883872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:57.884430 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:57.884710 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:57.884992 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] PATH_INFO: `/v3/users/53df6be4d82048819ab0cab5dd6b7947` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:57.886117 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=53df6be4d82048819ab0cab5dd6b7947)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:57.888131 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:58.164952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53df6be4d82048819ab0cab5dd6b7947'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.165532 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53df6be4d82048819ab0cab5dd6b7947'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.177288 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53df6be4d82048819ab0cab5dd6b7947'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.184061 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53df6be4d82048819ab0cab5dd6b7947'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.184682 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b97bb6ee-3f8d-459b-841d-216a9fd9c6a4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53df6be4d82048819ab0cab5dd6b7947'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.258347 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1303/2611] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:39:57 2022] DELETE /identity/v3/users/53df6be4d82048819ab0cab5dd6b7947 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:58.640650 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:58.642201 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:58.657228 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94864f0d-e5b6-4bac-8406-604a7764b763 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:58.714444 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:58.715016 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:58.715296 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:58.715563 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] PATH_INFO: `/v3/projects/5f728147c0f743ab980508bdd0aa4797` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:58.716668 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f728147c0f743ab980508bdd0aa4797)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:58.718859 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:58.848143 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f728147c0f743ab980508bdd0aa4797', 'request_id': 'req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.850608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:58.938775 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f728147c0f743ab980508bdd0aa4797', 'request_id': 'req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.945205 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f728147c0f743ab980508bdd0aa4797', 'request_id': 'req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.956442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] The token cache is being invalidate because project 5f728147c0f743ab980508bdd0aa4797 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:39:58.956733 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.957361 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-cdc6e63b-aaa0-4a46-bf7c-41587afda9f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:39:58.958874 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1309/2612] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:39:58 2022] DELETE /identity/v3/projects/5f728147c0f743ab980508bdd0aa4797 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:58.982721 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ede45a-5f98-4c65-a1fd-274bcd762447 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:58.983211 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ede45a-5f98-4c65-a1fd-274bcd762447 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:58.983302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ede45a-5f98-4c65-a1fd-274bcd762447 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.003681 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b5ede45a-5f98-4c65-a1fd-274bcd762447 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:59.004879 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b5ede45a-5f98-4c65-a1fd-274bcd762447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:59.047663 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:39:59.061750 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1304/2613] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:39:59.071190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed215af-6b11-4877-a92c-12aa1cf0e8e5 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.071475 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed215af-6b11-4877-a92c-12aa1cf0e8e5 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.071751 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed215af-6b11-4877-a92c-12aa1cf0e8e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.091934 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-fed215af-6b11-4877-a92c-12aa1cf0e8e5 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:59.093140 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-fed215af-6b11-4877-a92c-12aa1cf0e8e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:59.101485 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1310/2614] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:59.105664 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ede45a-5f98-4c65-a1fd-274bcd762447 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:59.115737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ede45a-5f98-4c65-a1fd-274bcd762447 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:59.120637 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:59.121319 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.121604 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.121872 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.122515 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:39:59.122815 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:59.125034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:59.136878 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1305/2615] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:39:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:39:59.141169 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed215af-6b11-4877-a92c-12aa1cf0e8e5 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:59.150421 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed215af-6b11-4877-a92c-12aa1cf0e8e5 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:59.156009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943b836a-bf0e-4432-a0a5-a5eb88da013f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:59.156009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943b836a-bf0e-4432-a0a5-a5eb88da013f admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.156009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943b836a-bf0e-4432-a0a5-a5eb88da013f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.156336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943b836a-bf0e-4432-a0a5-a5eb88da013f admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.156777 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943b836a-bf0e-4432-a0a5-a5eb88da013f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:59.158711 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943b836a-bf0e-4432-a0a5-a5eb88da013f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:59.281968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1311/2616] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:39:59 2022] POST /identity/v3/projects => generated 375 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:59.285922 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:59.296156 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff9c225-931f-4bb2-b38b-f38fe29b32cd admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:59.351616 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7e89a5-fafe-4a6e-8ab5-03e54510c0a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:59.352208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7e89a5-fafe-4a6e-8ab5-03e54510c0a2 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.352484 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7e89a5-fafe-4a6e-8ab5-03e54510c0a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.352754 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7e89a5-fafe-4a6e-8ab5-03e54510c0a2 admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.353376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7e89a5-fafe-4a6e-8ab5-03e54510c0a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:59.355341 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7e89a5-fafe-4a6e-8ab5-03e54510c0a2 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:59.453988 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1306/2617] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:39:59 2022] POST /identity/v3/users => generated 288 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:39:59.458202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943b836a-bf0e-4432-a0a5-a5eb88da013f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:59.468975 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943b836a-bf0e-4432-a0a5-a5eb88da013f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:59.474144 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b42ffa2-3f9b-4eaf-9be0-16a8440697fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:59.474765 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b42ffa2-3f9b-4eaf-9be0-16a8440697fa admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.475087 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b42ffa2-3f9b-4eaf-9be0-16a8440697fa admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.475364 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b42ffa2-3f9b-4eaf-9be0-16a8440697fa admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.476012 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b42ffa2-3f9b-4eaf-9be0-16a8440697fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:59.477617 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b42ffa2-3f9b-4eaf-9be0-16a8440697fa admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:59.488126 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1312/2618] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:59.491845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7e89a5-fafe-4a6e-8ab5-03e54510c0a2 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:59.501723 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7e89a5-fafe-4a6e-8ab5-03e54510c0a2 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:59.506475 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f0247b-d2a4-4794-8481-09fa31e40cda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:59.507098 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0247b-d2a4-4794-8481-09fa31e40cda admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.507376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0247b-d2a4-4794-8481-09fa31e40cda admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.507642 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f0247b-d2a4-4794-8481-09fa31e40cda admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.508276 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f0247b-d2a4-4794-8481-09fa31e40cda admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:59.509845 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f0247b-d2a4-4794-8481-09fa31e40cda admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:59.520235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1307/2619] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:39:59 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:39:59.524147 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b42ffa2-3f9b-4eaf-9be0-16a8440697fa admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:59.534579 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b42ffa2-3f9b-4eaf-9be0-16a8440697fa admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:59.539321 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:59.539960 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.540237 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.540508 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] PATH_INFO: `/v3/projects/87766696ff734b9c9892bbd1371142fc/users/ea0c6ef174be43119c4a32f61f67584f/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.555334 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=87766696ff734b9c9892bbd1371142fc, user_id=ea0c6ef174be43119c4a32f61f67584f, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:59.557941 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:59.647387 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1313/2620] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:39:59 2022] PUT /identity/v3/projects/87766696ff734b9c9892bbd1371142fc/users/ea0c6ef174be43119c4a32f61f67584f/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:39:59.657794 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcedb22-06ec-46e1-944e-a63063700c8b None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.658086 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcedb22-06ec-46e1-944e-a63063700c8b None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.658357 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcedb22-06ec-46e1-944e-a63063700c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.687821 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-6fcedb22-06ec-46e1-944e-a63063700c8b None None] MFA Rules not processed for user `ea0c6ef174be43119c4a32f61f67584f`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:39:59.689058 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-6fcedb22-06ec-46e1-944e-a63063700c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:59.752235 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1308/2621] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:39:59 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:39:59.764159 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:39:59.765826 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:39:59.821660 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6fcd9a-1857-407b-9a14-46338869fa4c admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:39:59.825251 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399ab4a8-1ecd-473f-8106-688156a205b5 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:39:59.825842 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ab4a8-1ecd-473f-8106-688156a205b5 service glance] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:39:59.826116 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ab4a8-1ecd-473f-8106-688156a205b5 service glance] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:39:59.826385 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ab4a8-1ecd-473f-8106-688156a205b5 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:39:59.834447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399ab4a8-1ecd-473f-8106-688156a205b5 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:39:59.835900 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399ab4a8-1ecd-473f-8106-688156a205b5 service glance] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:39:59.844721 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1314/2622] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:39:59 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:40:09.749443 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=76,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:40:09.840828 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:DELETE=3,keystone:SELECT=59,keystone:INSERT=2 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:40:19.160850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcedb22-06ec-46e1-944e-a63063700c8b None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:19.223772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcedb22-06ec-46e1-944e-a63063700c8b None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:19.228445 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:19.229062 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:19.229340 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:19.229609 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] PATH_INFO: `/v3/users/ea0c6ef174be43119c4a32f61f67584f` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:19.230929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea0c6ef174be43119c4a32f61f67584f)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:19.232907 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:19.446757 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea0c6ef174be43119c4a32f61f67584f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:19.447455 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea0c6ef174be43119c4a32f61f67584f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:19.460180 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea0c6ef174be43119c4a32f61f67584f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:19.467233 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea0c6ef174be43119c4a32f61f67584f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:19.467868 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea0c6ef174be43119c4a32f61f67584f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:19.524950 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1309/2623] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:40:19 2022] DELETE /identity/v3/users/ea0c6ef174be43119c4a32f61f67584f => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:40:19.529482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a67a48-2c12-42a3-b074-a8393780a80b None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:19.530006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a67a48-2c12-42a3-b074-a8393780a80b None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:19.530105 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a67a48-2c12-42a3-b074-a8393780a80b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:19.549380 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-57a67a48-2c12-42a3-b074-a8393780a80b None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:40:19.550562 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-57a67a48-2c12-42a3-b074-a8393780a80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:19.608074 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1315/2624] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:40:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:40:19.618321 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:19.619743 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:19.676396 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ad0256-df02-410d-b678-5febbbb9c7fc admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:19.680471 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eade967-1132-4a65-9682-87a29220ff9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:19.681060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eade967-1132-4a65-9682-87a29220ff9d service neutron] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:19.681335 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eade967-1132-4a65-9682-87a29220ff9d service neutron] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:19.681605 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eade967-1132-4a65-9682-87a29220ff9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:19.690291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eade967-1132-4a65-9682-87a29220ff9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:19.691734 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eade967-1132-4a65-9682-87a29220ff9d service neutron] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:19.700958 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1310/2625] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:40:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:40:20.309529 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a67a48-2c12-42a3-b074-a8393780a80b None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:20.310989 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-57a67a48-2c12-42a3-b074-a8393780a80b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:20.323437 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a67a48-2c12-42a3-b074-a8393780a80b None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:20.328374 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:20.337899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:20.337899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:20.337899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] PATH_INFO: `/v3/projects/87766696ff734b9c9892bbd1371142fc` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:20.337899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=87766696ff734b9c9892bbd1371142fc)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:20.337899 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:20.453033 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87766696ff734b9c9892bbd1371142fc', 'request_id': 'req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:20.521076 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87766696ff734b9c9892bbd1371142fc', 'request_id': 'req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:20.527148 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87766696ff734b9c9892bbd1371142fc', 'request_id': 'req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:20.538736 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] The token cache is being invalidate because project 87766696ff734b9c9892bbd1371142fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:40:20.539336 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:20.539704 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-1c8ca9fc-6217-429f-8611-f0d97e2f32e6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:20.541172 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1316/2626] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:40:20 2022] DELETE /identity/v3/projects/87766696ff734b9c9892bbd1371142fc => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:40:20.545952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22c687-2080-408d-b2a9-6f3226892c84 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:20.548573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22c687-2080-408d-b2a9-6f3226892c84 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:20.548573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a22c687-2080-408d-b2a9-6f3226892c84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:20.565895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2a22c687-2080-408d-b2a9-6f3226892c84 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:40:20.567207 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2a22c687-2080-408d-b2a9-6f3226892c84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:20.584048 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:40:20.626362 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1311/2627] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:40:20.634194 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56191f03-bc8a-49d1-91bd-a3d47009cb51 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:20.634478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56191f03-bc8a-49d1-91bd-a3d47009cb51 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:20.634758 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56191f03-bc8a-49d1-91bd-a3d47009cb51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:20.655932 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-56191f03-bc8a-49d1-91bd-a3d47009cb51 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:40:20.657264 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-56191f03-bc8a-49d1-91bd-a3d47009cb51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:20.666072 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1317/2628] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:40:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:40:20.670226 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a22c687-2080-408d-b2a9-6f3226892c84 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:20.680459 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a22c687-2080-408d-b2a9-6f3226892c84 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:20.685219 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:20.688643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:20.688643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:20.688643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] PATH_INFO: `/v3/domains` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:20.688643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83021) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:40:20.688643 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:20.689210 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:20.700094 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1312/2629] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:40:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:40:20.704256 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56191f03-bc8a-49d1-91bd-a3d47009cb51 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:20.714378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56191f03-bc8a-49d1-91bd-a3d47009cb51 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:20.719010 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad7f488-cdcc-41bc-94c0-5fc9d4002bbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:20.719602 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7f488-cdcc-41bc-94c0-5fc9d4002bbd admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:20.719879 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7f488-cdcc-41bc-94c0-5fc9d4002bbd admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:20.720151 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad7f488-cdcc-41bc-94c0-5fc9d4002bbd admin admin] PATH_INFO: `/v3/projects` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:20.720782 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad7f488-cdcc-41bc-94c0-5fc9d4002bbd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:20.722666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ad7f488-cdcc-41bc-94c0-5fc9d4002bbd admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:20.802304 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1318/2630] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:40:20 2022] POST /identity/v3/projects => generated 367 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:40:20.807043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:20.816896 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31afebcc-808f-43d2-9270-41c4543e66be admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:20.872816 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aebc1ae-9e97-42a6-a4e9-e42b7c93ce8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:20.873414 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebc1ae-9e97-42a6-a4e9-e42b7c93ce8d admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:20.873700 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebc1ae-9e97-42a6-a4e9-e42b7c93ce8d admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:20.873973 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aebc1ae-9e97-42a6-a4e9-e42b7c93ce8d admin admin] PATH_INFO: `/v3/users` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:20.874590 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aebc1ae-9e97-42a6-a4e9-e42b7c93ce8d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:20.876525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aebc1ae-9e97-42a6-a4e9-e42b7c93ce8d admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:20.958320 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1313/2631] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:40:20 2022] POST /identity/v3/users => generated 284 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 19 22:40:20.963367 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad7f488-cdcc-41bc-94c0-5fc9d4002bbd admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:20.973738 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad7f488-cdcc-41bc-94c0-5fc9d4002bbd admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:20.978513 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbdfff0-255c-4059-ae66-d3ec13fbdc22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:20.980813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbdfff0-255c-4059-ae66-d3ec13fbdc22 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:20.980813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbdfff0-255c-4059-ae66-d3ec13fbdc22 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:20.980813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbdfff0-255c-4059-ae66-d3ec13fbdc22 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:20.980813 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbdfff0-255c-4059-ae66-d3ec13fbdc22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:20.981945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbdfff0-255c-4059-ae66-d3ec13fbdc22 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:20.992181 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1319/2632] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:40:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:40:20.996083 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aebc1ae-9e97-42a6-a4e9-e42b7c93ce8d admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:21.005991 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aebc1ae-9e97-42a6-a4e9-e42b7c93ce8d admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:21.011056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a04139d2-4f87-49fe-8c08-347bf2a80c48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:21.011625 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04139d2-4f87-49fe-8c08-347bf2a80c48 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:21.011902 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04139d2-4f87-49fe-8c08-347bf2a80c48 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:21.012172 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a04139d2-4f87-49fe-8c08-347bf2a80c48 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:21.012808 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04139d2-4f87-49fe-8c08-347bf2a80c48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:21.014381 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a04139d2-4f87-49fe-8c08-347bf2a80c48 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:21.025493 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1314/2633] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:40:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:40:21.029430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbdfff0-255c-4059-ae66-d3ec13fbdc22 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:21.039987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbdfff0-255c-4059-ae66-d3ec13fbdc22 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:21.044585 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:21.045231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] REQUEST_METHOD: `PUT` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:21.045518 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:21.045792 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] PATH_INFO: `/v3/projects/4749cb430bf44a3183fdbb883d42efa9/users/a49a03c829f64ff882bc9ae4ad84063d/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:21.060163 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] RBAC: Authorizing `identity:create_grant(project_id=4749cb430bf44a3183fdbb883d42efa9, user_id=a49a03c829f64ff882bc9ae4ad84063d, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:21.062805 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:21.124312 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1320/2634] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:40:21 2022] PUT /identity/v3/projects/4749cb430bf44a3183fdbb883d42efa9/users/a49a03c829f64ff882bc9ae4ad84063d/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:40:21.136159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0853873-2df2-4ebe-b483-3156f2286f9a None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:21.136451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0853873-2df2-4ebe-b483-3156f2286f9a None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:21.136730 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0853873-2df2-4ebe-b483-3156f2286f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:21.164339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-b0853873-2df2-4ebe-b483-3156f2286f9a None None] MFA Rules not processed for user `a49a03c829f64ff882bc9ae4ad84063d`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:40:21.165541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b0853873-2df2-4ebe-b483-3156f2286f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:21.227916 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1315/2635] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:40:21 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:40:21.239114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:21.240788 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:21.263136 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:40:21.297546 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4afad361-012b-4df8-9006-a439a284042d admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:21.301128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67634275-55d6-4244-9164-b48aaf607cbb service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9ebc359d9bf54edcb9634afc31cf188a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:21.301722 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67634275-55d6-4244-9164-b48aaf607cbb service glance] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:21.301995 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67634275-55d6-4244-9164-b48aaf607cbb service glance] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:21.302259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67634275-55d6-4244-9164-b48aaf607cbb service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:21.310810 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67634275-55d6-4244-9164-b48aaf607cbb service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:21.312259 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67634275-55d6-4244-9164-b48aaf607cbb service glance] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:21.321447 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1321/2636] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:40:21 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:40:31.225174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:40:31.355391 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=17 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:40:48.758564 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0853873-2df2-4ebe-b483-3156f2286f9a None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:48.759967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b0853873-2df2-4ebe-b483-3156f2286f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:48.844736 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0853873-2df2-4ebe-b483-3156f2286f9a None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:48.849697 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d94fdf4-b309-421a-b429-15671377cb61 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:48.850314 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d94fdf4-b309-421a-b429-15671377cb61 service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:48.850573 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d94fdf4-b309-421a-b429-15671377cb61 service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:48.850840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d94fdf4-b309-421a-b429-15671377cb61 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:48.852568 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-8d94fdf4-b309-421a-b429-15671377cb61 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:48.863647 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d94fdf4-b309-421a-b429-15671377cb61 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:48.865212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d94fdf4-b309-421a-b429-15671377cb61 service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:48.874929 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1316/2637] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:40:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:40:58.567686 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67634275-55d6-4244-9164-b48aaf607cbb service glance] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:58.630478 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67634275-55d6-4244-9164-b48aaf607cbb service glance] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:58.635702 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:58.636353 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:58.636635 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:58.636909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] PATH_INFO: `/v3/users/a49a03c829f64ff882bc9ae4ad84063d` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:58.638035 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] RBAC: Authorizing `identity:delete_user(user_id=a49a03c829f64ff882bc9ae4ad84063d)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:58.640014 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:58.817168 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a49a03c829f64ff882bc9ae4ad84063d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:58.870672 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=39 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:40:58.890923 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a49a03c829f64ff882bc9ae4ad84063d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:58.903692 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a49a03c829f64ff882bc9ae4ad84063d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:58.910769 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a49a03c829f64ff882bc9ae4ad84063d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:58.911430 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a49a03c829f64ff882bc9ae4ad84063d'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:58.918131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1322/2638] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:40:58 2022] DELETE /identity/v3/users/a49a03c829f64ff882bc9ae4ad84063d => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:40:58.922557 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3632c5e9-8a33-4724-9710-a1d4ded9b492 None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:58.922840 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3632c5e9-8a33-4724-9710-a1d4ded9b492 None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:58.923142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3632c5e9-8a33-4724-9710-a1d4ded9b492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:58.943014 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-3632c5e9-8a33-4724-9710-a1d4ded9b492 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:40:58.944208 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3632c5e9-8a33-4724-9710-a1d4ded9b492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:59.001416 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1317/2639] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:40:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 19 22:40:59.011542 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:59.012942 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:59.068482 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b046a224-368f-4cb4-9a5c-59639599633b admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:59.072573 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1886288f-a8eb-47bc-9df7-a76aca2e7d59 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:59.073171 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886288f-a8eb-47bc-9df7-a76aca2e7d59 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:59.073448 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886288f-a8eb-47bc-9df7-a76aca2e7d59 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:59.073717 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1886288f-a8eb-47bc-9df7-a76aca2e7d59 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:59.082708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1886288f-a8eb-47bc-9df7-a76aca2e7d59 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:59.084173 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1886288f-a8eb-47bc-9df7-a76aca2e7d59 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:59.094496 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1323/2640] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:40:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:40:59.453106 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3632c5e9-8a33-4724-9710-a1d4ded9b492 None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:59.454541 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-3632c5e9-8a33-4724-9710-a1d4ded9b492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:59.466886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3632c5e9-8a33-4724-9710-a1d4ded9b492 None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:59.471451 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:59.472081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:59.472370 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:59.472624 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] PATH_INFO: `/v3/projects/4749cb430bf44a3183fdbb883d42efa9` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:59.473733 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4749cb430bf44a3183fdbb883d42efa9)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:59.475775 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:59.571353 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4749cb430bf44a3183fdbb883d42efa9', 'request_id': 'req-d27d1b4e-2186-403d-bb70-bce53b36a8f6'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:59.577601 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4749cb430bf44a3183fdbb883d42efa9', 'request_id': 'req-d27d1b4e-2186-403d-bb70-bce53b36a8f6'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:59.583270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4749cb430bf44a3183fdbb883d42efa9', 'request_id': 'req-d27d1b4e-2186-403d-bb70-bce53b36a8f6'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:59.644481 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] The token cache is being invalidate because project 4749cb430bf44a3183fdbb883d42efa9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:40:59.644771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:59.645392 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-d27d1b4e-2186-403d-bb70-bce53b36a8f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:40:59.646877 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1318/2641] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:40:59 2022] DELETE /identity/v3/projects/4749cb430bf44a3183fdbb883d42efa9 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:40:59.651953 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21848c91-6ac1-4c7e-befe-994f1c72cda2 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:59.652415 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21848c91-6ac1-4c7e-befe-994f1c72cda2 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:59.652531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21848c91-6ac1-4c7e-befe-994f1c72cda2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:59.672350 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-21848c91-6ac1-4c7e-befe-994f1c72cda2 None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:40:59.673604 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-21848c91-6ac1-4c7e-befe-994f1c72cda2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:59.690589 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:40:59.731287 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1324/2642] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:40:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:40:59.739162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d496e84-54b1-4689-8222-75ec33ef995f None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:59.743112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d496e84-54b1-4689-8222-75ec33ef995f None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:59.743112 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d496e84-54b1-4689-8222-75ec33ef995f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:59.760299 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-2d496e84-54b1-4689-8222-75ec33ef995f None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:40:59.761490 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-2d496e84-54b1-4689-8222-75ec33ef995f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:40:59.769327 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1319/2643] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:40:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:40:59.773530 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21848c91-6ac1-4c7e-befe-994f1c72cda2 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:59.783125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21848c91-6ac1-4c7e-befe-994f1c72cda2 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:59.787878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:59.789006 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:59.789289 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:59.789928 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] PATH_INFO: `/v3/domains` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:59.790192 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] RBAC: Adding query filter params (name=Default) {{(pid=83022) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 19 22:40:59.790492 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:59.792128 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:59.804749 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1325/2644] 10.222.0.20 () {58 vars in 1153 bytes} [Wed Oct 19 22:40:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 19 22:40:59.809054 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d496e84-54b1-4689-8222-75ec33ef995f None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:59.818427 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d496e84-54b1-4689-8222-75ec33ef995f None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:59.823162 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16cde64-c238-44f1-ab3e-bd7d4a795c80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:59.823756 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16cde64-c238-44f1-ab3e-bd7d4a795c80 admin admin] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:59.824043 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16cde64-c238-44f1-ab3e-bd7d4a795c80 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:59.824318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16cde64-c238-44f1-ab3e-bd7d4a795c80 admin admin] PATH_INFO: `/v3/projects` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:59.824952 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16cde64-c238-44f1-ab3e-bd7d4a795c80 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:59.826842 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16cde64-c238-44f1-ab3e-bd7d4a795c80 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:40:59.897498 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1320/2645] 10.222.0.20 () {60 vars in 1153 bytes} [Wed Oct 19 22:40:59 2022] POST /identity/v3/projects => generated 373 bytes in 89 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:40:59.901647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:40:59.911626 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6b9116-e35d-40ab-b2d5-4d17aaab17cf admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:40:59.966414 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6adb9e-6e96-45b5-9729-5bc084549ddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:40:59.967028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6adb9e-6e96-45b5-9729-5bc084549ddd admin admin] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:40:59.967327 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6adb9e-6e96-45b5-9729-5bc084549ddd admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:40:59.967595 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6adb9e-6e96-45b5-9729-5bc084549ddd admin admin] PATH_INFO: `/v3/users` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:40:59.968222 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6adb9e-6e96-45b5-9729-5bc084549ddd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:40:59.970125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6adb9e-6e96-45b5-9729-5bc084549ddd admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:00.045861 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1326/2646] 10.222.0.20 () {60 vars in 1144 bytes} [Wed Oct 19 22:40:59 2022] POST /identity/v3/users => generated 287 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 19 22:41:00.049914 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16cde64-c238-44f1-ab3e-bd7d4a795c80 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:00.060978 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16cde64-c238-44f1-ab3e-bd7d4a795c80 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:00.065861 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab1a6f8-a5bf-4bd9-9a0e-3bee9e794c49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:00.066457 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab1a6f8-a5bf-4bd9-9a0e-3bee9e794c49 admin admin] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:00.066737 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab1a6f8-a5bf-4bd9-9a0e-3bee9e794c49 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:00.067039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab1a6f8-a5bf-4bd9-9a0e-3bee9e794c49 admin admin] PATH_INFO: `/v3/roles` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:00.067679 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab1a6f8-a5bf-4bd9-9a0e-3bee9e794c49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:00.069272 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab1a6f8-a5bf-4bd9-9a0e-3bee9e794c49 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:00.079034 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1321/2647] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:41:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:41:00.085801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6adb9e-6e96-45b5-9729-5bc084549ddd admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:00.093221 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6adb9e-6e96-45b5-9729-5bc084549ddd admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:00.097882 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37439b4-5a62-46d7-8d5d-147b664d4935 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:00.098494 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37439b4-5a62-46d7-8d5d-147b664d4935 admin admin] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:00.098771 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37439b4-5a62-46d7-8d5d-147b664d4935 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:00.099079 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37439b4-5a62-46d7-8d5d-147b664d4935 admin admin] PATH_INFO: `/v3/roles` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:00.099718 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37439b4-5a62-46d7-8d5d-147b664d4935 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:00.101909 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37439b4-5a62-46d7-8d5d-147b664d4935 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:00.112261 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1327/2648] 10.222.0.20 () {58 vars in 1122 bytes} [Wed Oct 19 22:41:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 19 22:41:00.116194 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab1a6f8-a5bf-4bd9-9a0e-3bee9e794c49 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:00.126623 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab1a6f8-a5bf-4bd9-9a0e-3bee9e794c49 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:00.131171 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:00.131797 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] REQUEST_METHOD: `PUT` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:00.132074 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:00.132347 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] PATH_INFO: `/v3/projects/d31256a871ac463c87abaa6e497c5d55/users/b8e348b315b74e178fd5982026ada799/roles/974299ae4a414939a4f7e186921d4569` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:00.146302 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d31256a871ac463c87abaa6e497c5d55, user_id=b8e348b315b74e178fd5982026ada799, role_id=974299ae4a414939a4f7e186921d4569)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:00.149509 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:00.222376 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1322/2649] 10.222.0.20 () {60 vars in 1483 bytes} [Wed Oct 19 22:41:00 2022] PUT /identity/v3/projects/d31256a871ac463c87abaa6e497c5d55/users/b8e348b315b74e178fd5982026ada799/roles/974299ae4a414939a4f7e186921d4569 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 19 22:41:00.234378 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6ab1d2-fd89-4197-995d-62c3cb8408b4 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:00.234753 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6ab1d2-fd89-4197-995d-62c3cb8408b4 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:00.235131 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a6ab1d2-fd89-4197-995d-62c3cb8408b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:00.263505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-2a6ab1d2-fd89-4197-995d-62c3cb8408b4 None None] MFA Rules not processed for user `b8e348b315b74e178fd5982026ada799`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:41:00.264708 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-2a6ab1d2-fd89-4197-995d-62c3cb8408b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:00.327779 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1328/2650] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:41:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:41:00.339771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:00.341402 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:00.363248 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:41:00.397246 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d743641f-4cde-4ba4-a798-4f681d5251d7 admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:00.401523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82aa315a-13a9-4350-ab6e-074ecec8b20f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:00.402056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aa315a-13a9-4350-ab6e-074ecec8b20f service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:00.402333 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aa315a-13a9-4350-ab6e-074ecec8b20f service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:00.402600 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82aa315a-13a9-4350-ab6e-074ecec8b20f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:00.410722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aa315a-13a9-4350-ab6e-074ecec8b20f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:00.412177 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82aa315a-13a9-4350-ab6e-074ecec8b20f service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:00.421395 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1323/2651] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:41:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:41:00.445666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6ab1d2-fd89-4197-995d-62c3cb8408b4 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:00.465027 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:41:00.506680 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a6ab1d2-fd89-4197-995d-62c3cb8408b4 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:00.512184 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:00.512801 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:00.513137 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:00.513418 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] PATH_INFO: `/v3/users/b8e348b315b74e178fd5982026ada799` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:00.514531 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8e348b315b74e178fd5982026ada799)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:00.516479 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:00.685411 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8e348b315b74e178fd5982026ada799'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:00.776216 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8e348b315b74e178fd5982026ada799'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:00.788804 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8e348b315b74e178fd5982026ada799'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:00.795864 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8e348b315b74e178fd5982026ada799'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:00.796648 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8e348b315b74e178fd5982026ada799'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:00.802442 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1329/2652] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:41:00 2022] DELETE /identity/v3/users/b8e348b315b74e178fd5982026ada799 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:41:00.810056 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255637ba-c1fc-4f03-9fdc-0fddb7625efe None None] REQUEST_METHOD: `POST` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:00.810336 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255637ba-c1fc-4f03-9fdc-0fddb7625efe None None] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:00.810607 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255637ba-c1fc-4f03-9fdc-0fddb7625efe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:00.829658 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.auth.core [None req-255637ba-c1fc-4f03-9fdc-0fddb7625efe None None] MFA Rules not processed for user `4cc834c20ef040c19350af5fc47f4735`. Rule list: `[]` (Enabled: `True`). {{(pid=83021) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:41:00.830886 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-255637ba-c1fc-4f03-9fdc-0fddb7625efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:00.888572 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1324/2653] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:41:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:41:00.898190 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:00.899620 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:00.955568 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942d1fce-77e7-4300-98d9-c45653dc379f admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:00.959196 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759b81da-ddd4-496e-85b4-5460e2063845 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:00.959791 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759b81da-ddd4-496e-85b4-5460e2063845 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:00.960070 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759b81da-ddd4-496e-85b4-5460e2063845 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:00.960384 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759b81da-ddd4-496e-85b4-5460e2063845 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:00.969464 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759b81da-ddd4-496e-85b4-5460e2063845 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:00.970889 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759b81da-ddd4-496e-85b4-5460e2063845 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:00.980108 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1330/2654] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:41:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:41:01.375617 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255637ba-c1fc-4f03-9fdc-0fddb7625efe None None] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:01.377092 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-255637ba-c1fc-4f03-9fdc-0fddb7625efe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:01.390383 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255637ba-c1fc-4f03-9fdc-0fddb7625efe None None] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:01.395258 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:01.395883 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:01.396706 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:01.397231 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] PATH_INFO: `/v3/projects/d31256a871ac463c87abaa6e497c5d55` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:01.398463 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d31256a871ac463c87abaa6e497c5d55)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:01.400343 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:01.519122 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd31256a871ac463c87abaa6e497c5d55', 'request_id': 'req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:01.525359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd31256a871ac463c87abaa6e497c5d55', 'request_id': 'req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:01.530920 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd31256a871ac463c87abaa6e497c5d55', 'request_id': 'req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:01.592867 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] The token cache is being invalidate because project d31256a871ac463c87abaa6e497c5d55 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83021) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:41:01.593153 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:01.593772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:41:01.595289 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1325/2655] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:41:01 2022] DELETE /identity/v3/projects/d31256a871ac463c87abaa6e497c5d55 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:41:10.976499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=71,keystone:DELETE=10,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:41:11.589804 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=71,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:41:52.246878 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba11bc6-25a0-4f7a-9bd9-771931a2c3c7 None None] REQUEST_METHOD: `POST` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:52.247420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba11bc6-25a0-4f7a-9bd9-771931a2c3c7 None None] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:52.247514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba11bc6-25a0-4f7a-9bd9-771931a2c3c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:52.268831 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.auth.core [None req-1ba11bc6-25a0-4f7a-9bd9-771931a2c3c7 None None] MFA Rules not processed for user `690632c6d7494c22997ac41257ad32ff`. Rule list: `[]` (Enabled: `True`). {{(pid=83022) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 19 22:41:52.270009 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1ba11bc6-25a0-4f7a-9bd9-771931a2c3c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:52.327833 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1331/2656] 10.222.0.20 () {58 vars in 958 bytes} [Wed Oct 19 22:41:52 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 19 22:41:52.371139 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:52.372531 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:52.435523 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56a3639-cdc1-4ec3-bb39-06f1b71faf8f admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:52.439542 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'c065675e9a2f44f59168b9d574e289d5', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:52.440142 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:52.440422 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:52.440695 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:52.442680 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:52.513091 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:52.514525 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:52.525931 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1326/2657] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:41:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:41:52.807493 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba11bc6-25a0-4f7a-9bd9-771931a2c3c7 None None] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:52.809043 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-1ba11bc6-25a0-4f7a-9bd9-771931a2c3c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:52.865538 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba11bc6-25a0-4f7a-9bd9-771931a2c3c7 None None] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:52.869133 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:52.869725 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:52.870001 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:52.870271 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:52.872153 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:52.882199 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:52.883661 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:52.891936 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1332/2658] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:41:52 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:41:53.640215 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:53.641720 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:53.652793 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:53.658045 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:53.666028 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f66ef78-512e-48cd-afaf-31f236f67e1b service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:53.676114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:53.676730 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:53.677005 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:53.677278 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:53.679067 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:53.693188 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:53.694627 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:53.706294 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1333/2659] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:41:53 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 19 22:41:53.719767 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43093848-4cc6-43c5-b30e-ec4dfde2d4a1 service cinder] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:53.725222 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:53.725848 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:53.726116 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:53.726380 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:53.728159 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:53.739655 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:53.741291 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:53.752133 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1327/2660] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:41:53 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:41:57.327097 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:41:57.328655 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:57.343373 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5414b737-f173-41b2-8adf-584e71c61e3f service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:41:57.349341 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:41:57.349925 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:41:57.350202 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:41:57.350471 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:41:57.352281 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:41:57.365465 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:41:57.367666 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:41:57.382286 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1334/2661] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:41:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:42:03.747130 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=86 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:42:07.373999 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=63 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:42:07.662042 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:42:07.674530 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d962f35-9ef0-4f4f-89c3-089eb01e1d57 service placement] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:42:07.680653 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:42:07.681359 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:42:07.681641 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:42:07.681915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:42:07.683696 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:42:07.696771 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:42:07.698339 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:42:07.709202 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1328/2662] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:42:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:42:10.236514 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:42:10.246450 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039508da-6c8e-4b3a-8ce6-2c7a8762523f service nova] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:42:10.250112 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:42:10.250696 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:42:10.251004 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:42:10.251275 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:42:10.252973 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:42:10.336125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:42:10.337741 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:42:10.356968 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1335/2663] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:42:10 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:42:12.145932 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:42:12.158060 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25ba095-e9fd-4ea3-8fa0-e4846a0bb341 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:42:12.164039 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '35fc6d90245c4939bc7171201eb883bd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'ResellerAdmin', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:42:12.164622 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:42:12.164901 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:42:12.165174 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:42:12.166990 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:42:12.180462 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:42:12.182078 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:42:12.194260 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1329/2664] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:42:12 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:42:20.354303 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=43 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:42:22.188749 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=12 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:43:57.041483 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:43:57.060647 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094811b6-bf2a-4d6e-9e7d-632e17181f93 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:43:57.064231 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '2f705dc9d2e541f1821dda4b5c7b7709', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:43:57.064817 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:43:57.065089 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:43:57.065355 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:43:57.067113 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:43:57.129125 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:43:57.131114 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:43:57.140608 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1336/2665] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:43:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:44:00.736722 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:44:00.738245 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:44:00.754049 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af23b3d6-014b-4e05-8057-4cfc1bca01c5 service nova] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:44:00.760497 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:44:00.761282 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:44:00.761850 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:44:00.761967 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:44:00.763890 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:44:00.777279 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:44:00.778875 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:44:00.796795 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1330/2666] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:44:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:44:04.030945 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:44:04.043855 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b629b1bf-2358-4aa7-8e1c-c30a58820ab1 service neutron] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:44:04.048903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:44:04.049506 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] REQUEST_METHOD: `GET` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:44:04.049790 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:44:04.050068 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:44:04.051862 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:44:04.063903 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:44:04.065499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:44:04.078719 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1337/2667] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:44:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:44:04.164479 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:44:04.165976 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:44:04.180318 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc536b65-62b5-41a9-b434-8dec194c03ce service placement] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:44:04.185870 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:44:04.186500 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:44:04.186772 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:44:04.189212 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] PATH_INFO: `/v3/users/690632c6d7494c22997ac41257ad32ff` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:44:04.189798 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] RBAC: Authorizing `identity:delete_user(user_id=690632c6d7494c22997ac41257ad32ff)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:44:04.191785 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:44:04.356372 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '690632c6d7494c22997ac41257ad32ff'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.356953 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '690632c6d7494c22997ac41257ad32ff'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.370270 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '690632c6d7494c22997ac41257ad32ff'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.377259 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '690632c6d7494c22997ac41257ad32ff'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.377915 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '690632c6d7494c22997ac41257ad32ff'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.445296 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1331/2668] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:44:04 2022] DELETE /identity/v3/users/690632c6d7494c22997ac41257ad32ff => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:44:04.674098 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:44:04.687986 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb0b040-41b4-4f29-bef4-9032449db79b service placement] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:44:04.751095 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:44:04.751732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:44:04.752011 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:44:04.752285 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] PATH_INFO: `/v3/projects/9b8d72b42f384b4a9933c4648913857c` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:44:04.753420 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9b8d72b42f384b4a9933c4648913857c)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:44:04.755907 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:44:04.885469 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9b8d72b42f384b4a9933c4648913857c', 'request_id': 'req-5d97cfee-8657-42ac-a84c-3f126844c3e5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.953021 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9b8d72b42f384b4a9933c4648913857c', 'request_id': 'req-5d97cfee-8657-42ac-a84c-3f126844c3e5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.958852 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9b8d72b42f384b4a9933c4648913857c', 'request_id': 'req-5d97cfee-8657-42ac-a84c-3f126844c3e5'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.970570 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] The token cache is being invalidate because project 9b8d72b42f384b4a9933c4648913857c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:44:04.970857 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.971499 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:04.972987 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1338/2669] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:44:04 2022] DELETE /identity/v3/projects/9b8d72b42f384b4a9933c4648913857c => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:44:04.977076 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:44:04.978453 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:44:04.992586 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1280dd-7e2c-4074-8abd-38dc56d4374c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:44:05.049175 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:44:05.049800 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] REQUEST_METHOD: `DELETE` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:44:05.050081 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:44:05.050356 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] PATH_INFO: `/v3/users/812c54da9da240678f255ee5e192f5eb` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:44:05.051522 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] RBAC: Authorizing `identity:delete_user(user_id=812c54da9da240678f255ee5e192f5eb)` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:44:05.053433 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:44:05.235558 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '812c54da9da240678f255ee5e192f5eb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.236189 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '812c54da9da240678f255ee5e192f5eb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.246254 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=81,keystone:DELETE=18,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:44:05.248528 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '812c54da9da240678f255ee5e192f5eb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.255268 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '812c54da9da240678f255ee5e192f5eb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.255944 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.notifications [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '812c54da9da240678f255ee5e192f5eb'} {{(pid=83021) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.352537 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1332/2670] 10.222.0.20 () {58 vars in 1224 bytes} [Wed Oct 19 22:44:04 2022] DELETE /identity/v3/users/812c54da9da240678f255ee5e192f5eb => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:44:05.582505 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] Authenticating user token {{(pid=83022) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:44:05.584624 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.fernet_utils [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83022) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:44:05.593032 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:44:05.598625 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d97cfee-8657-42ac-a84c-3f126844c3e5 admin admin] Validating token access rules against request {{(pid=83022) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:44:05.655594 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4cc834c20ef040c19350af5fc47f4735', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '0fd8fb8e58c74c8cbe244194ce71b304', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83022) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:44:05.656214 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:44:05.656490 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:44:05.656778 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] PATH_INFO: `/v3/projects/787f90130cd842138b0c221e92d82f65` {{(pid=83022) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:44:05.657856 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=787f90130cd842138b0c221e92d82f65)` {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:44:05.659796 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] RBAC: Authorization granted {{(pid=83022) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:44:05.752832 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '787f90130cd842138b0c221e92d82f65', 'request_id': 'req-37fe6c01-9086-42ec-b46c-81133cb00dd1'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.792732 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '787f90130cd842138b0c221e92d82f65', 'request_id': 'req-37fe6c01-9086-42ec-b46c-81133cb00dd1'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.798653 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '787f90130cd842138b0c221e92d82f65', 'request_id': 'req-37fe6c01-9086-42ec-b46c-81133cb00dd1'} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.810880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] The token cache is being invalidate because project 787f90130cd842138b0c221e92d82f65 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=83022) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 19 22:44:05.810880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.810880 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG keystone.notifications [None req-37fe6c01-9086-42ec-b46c-81133cb00dd1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=83022) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 19 22:44:05.811935 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: [pid: 83022|app: 0|req: 1339/2671] 10.222.0.20 () {58 vars in 1233 bytes} [Wed Oct 19 22:44:05 2022] DELETE /identity/v3/projects/787f90130cd842138b0c221e92d82f65 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 19 22:44:15.404084 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:DELETE=2,keystone:SELECT=7,keystone:INSERT=1 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:44:15.806591 dv-cinder-master-3-iscsi devstack@keystone.service[83022]: DEBUG dbcounter [-] [83022] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=83022) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 19 22:45:52.738394 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] Authenticating user token {{(pid=83021) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 19 22:45:52.748552 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:45:52.809470 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9616aad6-3646-4c67-b107-a5a69461266c admin admin] Validating token access rules against request {{(pid=83021) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 19 22:45:52.867979 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc773de-ceca-4c0f-849c-0bda75308e24 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0a6d53677cb64aaba09e89eb527879cb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'efdf8a1e8219440c8d2a0e7395e2b1f4', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=83021) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 19 22:45:52.868579 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc773de-ceca-4c0f-849c-0bda75308e24 service placement] REQUEST_METHOD: `GET` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 19 22:45:52.868895 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc773de-ceca-4c0f-849c-0bda75308e24 service placement] SCRIPT_NAME: `/identity` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 19 22:45:52.869163 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc773de-ceca-4c0f-849c-0bda75308e24 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=83021) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 19 22:45:52.870897 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.fernet_utils [None req-edc773de-ceca-4c0f-849c-0bda75308e24 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=83021) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 19 22:45:52.881925 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc773de-ceca-4c0f-849c-0bda75308e24 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 19 22:45:52.884004 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc773de-ceca-4c0f-849c-0bda75308e24 service placement] RBAC: Authorization granted {{(pid=83021) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 19 22:45:52.893715 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: [pid: 83021|app: 0|req: 1333/2672] 10.222.0.20 () {60 vars in 1365 bytes} [Wed Oct 19 22:45:52 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 19 22:46:02.889478 dv-cinder-master-3-iscsi devstack@keystone.service[83021]: DEBUG dbcounter [-] [83021] Writing DB stats keystone:SELECT=58 {{(pid=83021) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}